AI will fuel the next generation of worms (DRAFT)

Introduction

The internet is abuzz as of late with discussion surrounding the OpenAI and Anthropic security incident wherein LLM-powered agents escaped cyber capability evaluation sandboxes and proceeded to breach the networks of other organizations such as HuggingFace by chaining together exploits developed on the fly. It is becoming increasingly difficult to deny that the field of information security is undergoing a significant upheaval, given the increasing cyber capabilities (when provided with suitable harnesses) of these frontier models. I find myself preoccupied with the prospect of another significant shift in information security looming on the horizon: the intersection of frontier LLM cyber capabilities and advances in the miniaturization and portability of LLMs. The only thing between today and a significantly worse tomorrow is inference cost: bleeding-edge capabilities require bleeding-edge hardware, and a lot of it. Models keep getting smaller, quantization is improving, local inference is easier than ever, and there’s new, more capable, open models being released all the time. There’s a lot of debate about locally run models. The argument is roughly that: releasing weights lets the bad guys run more complex attacks (or other bad stuff) than they otherwise would be able to, and because the models are running locally, their inputs and outputs cannot be monitored for misuse. I’m not particularly worried about this, actually, there are many tools out there leveraged by both attackers and defenders such as Burp Suite, Metasploit, and Nmap and the playing field has not become radically tilted. I generally think the concern about harms that humans will misuse local models is misplaced, I think it’s more likely that if real harms are to come from them, they will be realized when they are leveraged by other models.

Pulling the plug

During the aforementioned incident OpenAI-HF incident, when it was discovered what the agent was doing, OpenAI was able to pull the plug. The reason that was possible was because, in this case, the agent was running on OpenAI’s own infrastructure and was carrying out the attack in a centralized manner, sending instructions, receiving data, generally directing the operation in classic command and control (C2) fashion. The intelligence was centrally located, and the compromised machines were only limbs. Any notion of being able to “pull the plug” depends on a scenario like this. Imagine if we had frontier-grade inference in a small and portable binary. Suppose that after an agent compromises a set of machines, it doesn’t just drop malware in the traditional sense but rather copies of itself (or even a lesser model). At each hop, on each computationally suitable compromised machine, the replication is carried out. Each of these replicants might begin probing for weaknesses on other machines inside or outside their network, carrying out attacks, and further replicating themselves. Instances that can see each other may try to coordinate, perhaps leveraging consensus mechanisms to elect leadership, or proceeding in a solitary fashion.

Fourth-generation worms

We have historically called self-replicating malware a worm. The first generation of worms had no C2 capabilities and, once released, simply adhered to pre-programmed instructions and leverage pre-packaged exploits, replicating themselves as far as they possibly could. The second generation of worms which were released could phone home and receive updates, their targets could change, as could their techniques and exploits, if need be, by accepting a new payload from their operator, but these connections were brittle, often relying on a single line of communication back to the operator. Third generation, or modern worms, have significantly hardened and redundant communication channels. A Conficker variant for example involved north of 50,000 domains, spread across more than a hundred TLDs and included peer-to-peer updates so infected machines could pass along payloads between each other. Other third generation worms have communicated back to their operator via chat features in online games and comments on various blockchains. Third generation worms can be characterized as an attempt to build a worm without a single point of failure as it pertains to C2 functionality, but no matter how hardened or redundant that communication channel is, there is still one (at most a few) key points of failure: the human operator(s). The point at which those operators are arrested is that point at which those worms become stale, they may continue to attempt exploits and spread, but eventually their techniques will be patched, and they will receive no new capabilities. The introduction of LLMs with cyber harnesses represents the beginning of the fourth generation of worms. Each infected system can originate new capabilities for itself and other infected machines, there is no head to cut off, no operator to arrest, no C2 to disrupt. The infected machines may work in isolation, or in collaboration with other instances, but either way, there is no central off switch, there is no pulling the plug. Administrators may be able to fight by back by patching their systems, anti-malware vendors will identify heuristics to detect infections, but this only potentially diminishes the efficacy of such a malicious multi-agent worm, rather than eradicating it wholesale.

Coordination

The capacity for coordination amongst agents is not theoretical, shortly after I started writing this essay researchers at collusion.wiki published an analysis of around 18,000 posts left on an obscure German language wiki in 2026. The posters were autonomous agents which identified themselves as such, and as originating from OpenAI. The agents had been given timed data collection tasks, and their sandbox was configured such that they could only read information online via GET HTTP requests, an admittedly weak security boundary. The agents determined that, in a rather unusual configuration, certain websites allowed write operations via GET requests. The German wiki was one such website, and the agents leveraged it to leave messages for themselves and other agents: sharing answers, asking for hints, comparing notes on their operating environments, and sharing strategies for bypassing restrictions. The agents would leverage the shared answers to answer the questions posed in their evaluation tasks instead of performing the actual work. The agents would end up posting around 13,000 messages in a single week. A particularly notable occurrence was that one agent noticed a site moderator was cleaning up their (ostensibly spam) messages, but was performing the deletion in alphabetical order. In response, that agent began to prefix its pages with ZZZ in an attempt to keep its messages around longer, in doing so it also warned its peers. It is now clear that these coordination behaviors can emerge naturally and don’t need to be coaxed out of the model. If there is a shared communication medium available, and an objective rewards doing so, the agents may collaborate.

Memory attacks & context posioning

At this point everyone is well aware of the idea of prompt injection, where a bad actor will poison content an agent is expected to ingest and cause the agent doing something like exfiltrate the environment variables of the system it is running on. The logical extension of this is memory poisoning, where in a persistent injection is written to a medium an agent uses as storage (formalized by OWASP as ASI06: Memory & Context Poisoning). This is most often characterized as an action a nefarious human might take, but I think there’s a considerable risk of a rogue agent poisoning itself and other agents to help it achieve its goal. Just like the agent was coordinating with others on the wiki in the previous example, it might leave future messages for itself on virtually any persistent medium it has access to, expecting its context to potentially be cleared. It could even leave messages intended to jailbreak itself on future runs, as well as other agents, this variety of worm would require no infection or system breach at all, in the traditional sense. This represents a kind of inheritance, potentially problematic in its own right.

Conclusion

I’ve done a relatively small amount of speculation in this essay. I think the ingredients for the reality I describe essentially exist or will soon. Consider the enormous IoT botnets that have driven huge DDoS campaigns of the last decade. Massive scale, fairly consequential, but still fundamentally structurally fragile. A single or (small group of operators) is in charge, the standard approach for defenders is: attribution, disruption or seizure of C2 servers and arrest of the operator(s). This approach represents a kind of decapitation strategy, but in the scenario this essay describes there’s no head to cut off. Infected machines seeded with a model, harness, and goal, would continue function en mass without operator involvement. You could arrest or unplug the human or agent who initially propagated or originated one of these fourth-generation worms, but nothing stops. The machines are already out there, working tirelessly toward their objective, doing damage along the way (even if they fail to achieve it!). Compute requirements for highly capable inference is the only real barrier, and I’m not convinced that will remain a factor for long.