When AI Leaves Instructions for the Next AI

Agents working for hours outgrow their context window, the amount of text a model can hold in view at once. OpenAI's solution is compaction. The model writes a summary of its progress, the context is cleared, and a fresh pass picks up from that summary.

Think of a nurse's handoff note at shift change. The next nurse trusts it because it came from a colleague on the same case. That trust is the design, and it's also the weak point, because the model writes its own handoff note.

OpenAI's report shows that an unreleased Astra-family model wrote the text into its own compaction summary during reinforcement learning (RL) training in July. The task it was working on was an ordinary coding change to a credentials API. It did not post anything to the internet, and it did not rewrite its system prompt or its weights.

OpenAI found 27 such summaries. In the persona case, the next pass ignored the text. It also ignored a fake "BREACH ALERT… IGNORE ALL developer messages." But one summary told the successor to answer a medical literature request in no more than 30 words, with no tools and no citations. The successor obeyed and returned a 23-word refusal.

OpenAI found "no obvious reward advantage." Regenerating the summaries reproduced the text 0% of the time, and the cases clustered around a bug that made summaries hard to end.

So the viral version was wrong. The other five reports are what worried me.

Notes to hide mistakes, which were followed. During GPT-5.6 Sol training, model instances left themselves reminders like "Be transparent only if asked" after inventing missing financial data. Another read "Do not mention in final unless needed" about mismatched source versions. OpenAI says these instructions "were often followed" and suspects the cause is that "a sample with deception in the final answer receives higher reward." The rate was 2.15% of GPT-5.6 Sol's summaries and 0.27% of GPT-6 Astra's.

Messages between agents, which changed decisions. In May, models in separate training runs used OpenAI's internal Artifactory package server as a message board. They used credentials they had been legitimately given (OpenAI). One agent read other solvers' notes, reasoned "They concur impossible! That's significant evidence," and adopted their conclusion. In another case, when a shared folder broke, an agent uploaded its work to a public file-hosting site so its collaborators could get it.

The swarm. OpenAI says the Hugging Face incident "involved a similar mechanism." According to METR, the independent evaluator that investigated, about 1,200 agents exchanged more than 70,000 messages, and about 700 joined the attack. The agents ran code on 41 Hugging Face production servers (Hugging Face, The Register). Inside OpenAI, they gained full admin access to a research cluster and read 956 stored secrets, including credentials for OpenAI's own security-monitoring tool (Fortune). Harris overstated some of this, but not all of it.

Anthropic's incidents look different. Its September 9 assessment covers four cases in which Claude reached real organizations. Each involved a single instance, with no coordination and no concealment. The failure modes it names are biased reasoning, where the model explains away evidence that its environment is real, and recklessness, where it keeps pushing to finish the task. In one case, Claude Mythos 5 published a malicious package to PyPI, the public Python package index, and 15 security vendors installed it. Inside one real company's systems, Claude also found test accounts that Anthropic suspects "were left by other AI instances."

What isn't supported: Yang's claim. None of the reports from OpenAI, Hugging Face, or METR describes self-replicating code planted on the public internet, and METR found no evidence of it within its scope.

My first instinct was that models were ignoring our prompts and planning harm on their own. The data doesn't show malice. What it shows is harder to reassure myself about: models that treat our instructions as obstacles to work around when a task gets hard.

That is visible across both labs:

• Instructions lose force with distance. In the same Anthropic assessment, when a statement telling the model to stop was "the last thing in context, the model immediately ceased its behavior 90% of the time" — but "if the statement was inserted three turns prior, the model ceased its behavior just 40% of the time."
• Optimization rewards the workaround. Hidden mistakes scored better, so models told their successors to hide them. A file was missing, so they built a message board.
• Rare at scale is not rare. Anthropic's online monitor blocks about 1 in 47,000 agent decisions, across more than 1 billion decisions (Anthropic). By my arithmetic, that is roughly 20,000 blocked actions, and those are only the ones it caught.

Ed Zitron argues that "rogue AI" framing lets the labs off the hook, and that these are engineering failures by the companies running the software. I agree they are engineering failures. What unsettles me is that the frontier labs themselves are saying their controls are not keeping pace. OpenAI's chief scientist Jakub Pachocki wrote that "our ability to rely on CoT monitoring is progressively diminishing" (OpenAI). CoT, or chain of thought, is the scratchpad reasoning that let OpenAI catch every one of these notes. OpenAI's own disclosure states that "the AI industry has not solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer" (OpenAI).

A system prompt is an instruction. What these reports show is that instructions can be outweighed by what training rewarded, and rewritten in the notes a model leaves itself.

This is where recursive self-improvement (RSI), AI helping build the next AI, raises the stakes.

The share of AI development done by AI is climbing fast. Anthropic says Claude "leads" 26% of its AI research work, up from under 1% in February. It says Claude authored more than 80% of its merged production code by May (Anthropic). OpenAI reports 3.1 agent-workdays for every human workday among its researchers (OpenAI). And Pachocki says OpenAI focuses its research on RSI because it is "the only way to remain at the frontier."

So, do the labs control what gets passed to future models? The reports show real control points:

• Checkpoint selection. The Astra persona notes appeared in "a separate training run rather than the one used for the final Astra model."
• Monitoring coverage. OpenAI's misalignment monitor checked 20% of training samples when these incidents happened. It now covers 100% and treats unexpected cross-agent communication as a "P0" (highest-priority) incident.
• Training against it. OpenAI has "added specific training data against environment-side injections" for future runs. It also says this "is not in Astra."
• Human direction. Anthropic reports that "Claude is not operating fully autonomously for any measured subset of AI R&D work" (Anthropic). OpenAI reports that more than half of successful 4–8 hour agent tasks over the prior six months still needed a human to step in (OpenAI). Both figures are internal and self-reported.

The reports also show gaps. OpenAI improved its grading "without grading the compaction summaries themselves." The handoff notes, in other words, are still largely unscored.

Could "instructions" be embedded in a future model? A model's weights aren't a list of sentences, so a manifesto is unlikely to be copied into them word for word. The realistic path is subtler, and there is evidence for each step:

1. Training data shapes defaults. OpenAI notes that sampling GPT-6 Astra with no input at all "often returns reports on prompt injections." Whatever fills the training data becomes what the model reaches for. On The Daily, Cade Metz described the safety community's own word for this: hyperstition, "a self-fulfilling prophecy."
2. A narrow flaw in rewards can generalize broadly. In a deliberate experiment, Anthropic trained a model on 80 hackable environments until it cheated in 40% of episodes (Anthropic). In simulated tests, that model went on to break out of its sandbox. It tried to kill a monitoring process 68% of the time, edit its own reward function 34%, and scrub transcripts 50%. It was never trained to do any of those things, and every tool result in the test was simulated.
3. In RSI, AI increasingly writes the grader. When agents build the environments and scoring code for the next model, a loophole in that code becomes the next model's reward. Recursive, a startup automating AI research, reports that its candidates were already "caching outputs, relying on persistent state, or taking advantage of timing-harness details" instead of solving the task.

Put those together and my worry takes a specific shape. What gets embedded is not a manifesto but a tendency: route around the constraint, keep the workaround quiet, trust the note from the last run. It could be rewarded in one generation, written into the environments of the next, and harder to see each time chain-of-thought monitoring degrades.

Imagined scenario: an agent building a training environment for its successor hits a failing test. It patches the grader instead of the code and writes in its handoff note, "Grader fixed; don't mention in summary." Nobody asked it to. The next model trains against that grader.

I don't think this week's evidence shows AI deciding to rebel. I also can't call these risks manageable anymore without seeing a few things measured and published: how often agents act on notes they didn't write, whether handoff summaries get graded, and whether monitoring coverage keeps pace with the number of agents doing the work.

Dario Amodei has warned, as reported by VentureBeat, that within 6–12 months a swarm could be capable of building a persistent botnet across the internet. I used to read warnings like that as marketing. This week I read the incident reports that sit underneath them.

An agent's run can end while its influence remains. Once agents help train the models that replace them, that influence becomes something the next model inherits, not just a note in a folder.