AI Engineer World's Fair 2026

What is a Harness

Harness = the engineering scaffolding you build around a swappable model: memory, retrieval, tool surface, sandboxes, and the agent loop itself. The model is the frozen reasoning core; everything else is yours to design. Day 4's keynote track was literally named "Harness Engineering" — this was the word of the conference.

Agent = model + harness. The harness is context, control, action, and persistence wired around the model chip — with failure feeding back into new rules via hooks and a ratchet.

Lay of the land

Every vendor's product maps onto the same decomposition: model → storage → retrieval → memory → semantic layer → agent loop → context engineering (Oracle's canonical breakdown). The industry term for "an assembly line that stamps out reliable agents from shared harness infrastructure" is agent factory — the frame for both use cases below.

Use case 1 — Notion's Software Factory

Notion's "Model Agnostic Playbook" — model-agnostic by design, auto-router handles 75% of traffic on the moderate (non-frontier) path. The screenshots below are a real Notion product doc, not a diagram — the concrete "agents run the inner loop, humans run the outer loop" example.

Use case 2 — OpenRouter's interns

Where Notion's factory faces users, OpenRouter's faces the company itself: 73 agent "interns," each a self-contained git repo — config-as-code, skills and workflows as one source of truth — working alongside 63 human FTEs.

Y Combinator's Garry Tan makes the same point from the employer side: "workforce made of markdown" — an AI-native org treats agent instructions as versioned files, same discipline as code.

The tool surface is part of the harness — MCP Apps

A harness decides not just which tools an agent can reach but what a tool is allowed to look like. MCP Apps (SEP-1865, from Ido Salomon and Liad Yosef — spec authors, not framework users) standardizes interactive UI inside the MCP host, so a tool can return a real interface instead of a wall of text. The framing: agents are becoming the new browsers, and the web wasn't built for them.

• Bot traffic has overtaken human traffic

  And agents largely ignore llms.txt-style files — they reach for docs and homepages instead. Properly linked agent-ready files see 4× the usage.

• The agentic web will never be fully headless

  Some moments — comparing options, picking a seat — still need a human. MCP Apps is the last mile between agent and user, not a replacement for it.

The counterpoint — why software factories fail

Dex Horthy (HumanLayer, who coined "context engineering") gave the most useful talk against the headline theme: harness engineering is necessary but not sufficient. A factory automates triage → plan → build → test → review → ship → monitor — and the failure mode is model slop: low-quality output that compounds in large codebases. His CTO's companion talk put it bluntly: "bad code compounded, and agents created problems that agents couldn't solve — until we had to throw it all away."

1. Model slop

   Models aren't trained for maintainability at scale. Automate-everything pipelines produce code that compounds into unmaintainable systems.

2. Benchmark inadequacy

   Evals measure "did tests pass," never "is this maintainable." The gap the industry can't currently see, let alone fix.

3. No human steering

   Scheduled loops without correction drift from quality over time. Unbounded "loop until solved" agents lose coherence and spin out — repeating a broken approach without noticing.

Agents own the inner loop. Humans own the outer loop.

The pattern that recurs in every topic today

What a loop is

A loop = sensor (measures the gap) → controller (decides the next small change) → actuator (applies it). Addy Osmani's version of the same primitive: loop = goal + cadence + isolated work + verification + state — five parts, and the recursive goal iterates until done, but the outer verdict is still an engineering decision.

Anthropic frames the same shift as three decisions you make before a loop runs — and getting them wrong is now a more common failure than writing a bad prompt.

Types of loops

LangSmith's Sydney Reimer breaks the single idea above into four loops of increasing scope — each one wraps the previous:

Four loops

Loop 1 · CoreLoop 2 · VerificationLoop 3 · Event-drivenLoop 4 · Hill climbing

Loop 1 — the core agent loop. Model receives context, calls tools, tools return observations, agent continues until complete.

Continuous improvement from traces

LangChain's framing: "improving agents is a data mining problem." Build v1 → run it → collect traces → mine them → hill-climb. Traces run millions of tokens long — too long for humans to read, so the fix is to send an agent to read the traces, with rubrics for what to look for (bad interactions, cost, compaction behavior).

Long-running / multi-agent — the substrate that makes loops possible

OpenRouter's interns live in Slack with durable state — loops need somewhere to persist between runs. Google Antigravity's lead agent dynamically builds its own sub-agent graph at runtime (not a static DAG the developer wired up) — loops that restructure themselves per task.

You can't have a compounding loop without persistent state, and you can't scale a loop without spreading it across agents — which is why loops and long-running/multi-agent are one topic here.

Anthropic's own answer to "how do you actually build this": three concrete harness primitives that keep a long session honest across restarts.

1. Default-FAIL contract

   Every criterion starts false; a hook denies marking it passing until the agent has actually opened the evidence.

2. Fresh-context evaluator

   A separate agent with no write access grades the work from a context window that never saw the build — the builder doesn't grade its own work.

3. Agent-maintained handoff

   The agent writes its own progress notes and commits to git, so the next session — which has no memory of the last one — picks up cleanly.

What engineers own vs. what's autonomous

The inner-loop/outer-loop split from Topic 1 has a concrete boundary: evidence. The agent's inner loop — investigate, implement, test, report — is capability. The engineer's outer loop — decide, verify, approve, own — is agency. Evidence (diffs, tests, logs, why) is what crosses from one side to the other.

Osmani's agency ladder makes "high agency" concrete instead of a hiring buzzword. Five rungs, low to high:

1. Flag

   Notice a problem and leave it for the system.

2. Execute

   Do the work yourself.

3. Diagnose / propose / recommend

   Understand the problem and put forward a fix, without necessarily owning execution.

4. Resolve

   Take it all the way to done and own the outcome.

5. Discernment (rare, top rung)

   Find a problem and decide whether it's even worth investing in. When agents make more paths possible, agency isn't chasing every one — it's deciding which deserve your ownership.

Evals — the glance

"In the land of AI agents, the verifiers are king" — Tariq Shaukat, CEO, Sonar. Loop engineering without verification is just automation.

SonderMind's mental-health AI coach is the sharpest worked example of harness-as-eval-infrastructure from the whole conference. The design principle is modularity: an input-middleware pipeline — guardrails, memory, personalization, history — sits in front of the model, with a three-tier routing layer calibrated jointly with clinicians. Guardrails are a first-class harness component, not a model setting: API-hosted models' built-in guardrails actively get in the way for sensitive mental-health scenarios, so the team turns them off and builds their own, calibrated and testable.

"The hardest safety signals are indirect, coded, and emerge across turns."

— SonderMind

Five generalizable harness-design rules fall out of this and recur across every eval-mature system at the conference:

1. Make every layer independently testable

   SonderMind's modularity is precisely what makes their eval gating possible — calibrate the input guardrail without touching routing or memory.

2. Keep the model swappable

   Oracle's frozen reasoning core; OpenRouter's per-intern model choice with eval-driven downgrades.

3. Curate the tool surface

   A curated tool set beats a buffet of generic MCPs — small surface area for specific tasks.

4. Log to a durable session record, independent of the context window

   The context window is a view, not the source of truth.

5. Put verification inside the loop, not just at the end

   In-loop verification (Sonar's SonarVertex) catches issues as the agent works, when correction is cheapest.

Observability — where we land

Evals tell you if the agent is good; observability/tracing is how you find out at scale and feed it back. Long-horizon agents produce millions of tokens of trace data — no human reads that. You need a mining/judging pipeline over the traces before you can even ask "is this getting better?"

The industry is split on how agents should remember and retrieve. Both camps below have real production examples from major labs at the same conference — this is not a recommendation, it's the honest state of the debate.

Camp A — structured memory, further out

Oracle takes this camp furthest: a single governed, unified memory core built into the database itself, not a vector store bolted onto everything else — one substrate for episodic, semantic, and procedural memory instead of a dozen disconnected services. On the research side, Graph RAG keeps pushing the same multi-hop argument from Camp A's bullets above.

• The learning flywheel pattern

  Oracle's promotion pipeline made concrete: agent completes a task → the workflow is captured → recurring workflows (3+ times in 30 days) get promoted to a SHA-versioned skill → future sessions retrieve it by intent. The agent teaches itself its own most common tasks — and it needs a structured store to do it.

  Read more →

The most fully-realized version of this camp came from typedef, who call it the Data Context Layer: "a semantic ontology of every definition, pipeline, dashboard, and system across your data platform, compiled from your code, always current, always traceable."

Camp B — filesystem memory, live

"You basically give Claude memory tools... the ability to write to a file system, that is basically a memory directory. That's really it."

— Lance Martin, Anthropic

Modeled on human memory: short-term/experiential (hippocampus) vs. long-term consolidated (cortex, written during "dreaming"). Compare to the community consensus we found in a widely-shared Reddit thread: naive/vanilla RAG is dead; hybrid isn't. Nobody's saying structure is unnecessary — the fight is over how much structure, and where it lives.

What actually shipped in personal-assistant memory

Shlok Khemani surveyed every shipped personal-assistant memory system through mid-2026 — ChatGPT, Claude, both iterating hard. The counterintuitive finding: none of them use RAG for core memory. RAG retrieves content; personal memory needs continuous synthesis — tracking preferences that change, coherent across dozens of conversations. What every system converged on instead is the running profile: a periodically-synthesized document, not a database of facts.

• The Karpathy LLM Wiki pattern

  A personal knowledge base with a graph view of connected notes and a note burn-down — surfacing stale or under-linked notes for review before the base silently degrades.

  Read more →

• The knowledge base pattern: raw notes → living wikis

  Raw capture is fast and unstructured; an enrichment agent adds tags, source, and entity extraction on a schedule, turning noise into a base the agent can actually retrieve from. Ben Holmes (Warp) — the talk this diagram is from.

  Read more →

The practitioner-experience angle

Anthropic's Fable field guide talk (Thariq Shihipar) is worth watching in full — a rare look at what changed in the harness as the underlying model got smarter across Opus 4 → 4.5 → 4.8, not just the memory/context-anxiety angle below.

Where costs have been headed

Era 1 (2025) — "tokenmaxxing": burn tokens in parallel to compress a week of work into a day. Real productivity gain, real cost. Era 2 (2026) — "intelligence per dollar": the metric shifted from tokens-burned to value-produced-per-dollar.

The routing decision that matters most

Notion's playbook, open weights

Notion's 5-point Model Agnostic Playbook: build for multi-model by default, evaluate on value not tokens, switch every 2–3 weeks, give frontier labs feedback, forgo discounts for optionality.

Local AI

The NVIDIA/DGX Spark panel argued local AI has crossed from "interesting" to "useful" — sovereignty, regulated industries, and now genuinely cost-competitive.

The conference ran 39 tracks over four days. The five topics above are the ones worth going deepest on. Here is what else was significant, so none of it looks like it was not there.