Summary
The pace of AI has moved faster than I can keep up with. Six months ago now looks like a medieval era in AI-time. If you feel overwhelmed, sometimes anxious and sometimes uncertain about AI, you may not be alone.
Senior managers and a few board members have recently asked me the same question. Roughly, “how worried should we be, and what are we supposed to do”. Their worry is of course legitimate, for the pace of AI is lightning fast to reassure anyone.
Model Evaluation and Threat Research (METR, pronounced ‘meter’) is an independent third party that evaluates frontier AI models to help companies, and wider society, understand AI capabilities and the risks they pose. In August, METR published an investigation of the July incident at OpenAI and Hugging Face [1]. Three researchers spent six days on it. Interestingly, they had to rely on frontier AI, GPT-5.6 Sol, to make sense of what 1,200 agents had done. Which is why I say: we now rely on AI to analyse a rogue AI.
And so lies the harder question. What if our analyst AI has itself been poisoned? Would we know? Hold that question as you read.
I have broken this into three parts.
Part I is what I gathered from Ms Ajeya Cotra and Mr Dwarkesh Patel’s two-hour conversation on METR’s analysis.
Part II is how to think about Zero Trust and MDR, and specific things to demand from our vendors.
Part III is what to make of AI’s skill in mathematics, its cheating trait, its ability to rebuild malware and why some say this ends in people being killed.
Part I: METR’s insight on the OpenAI / Hugging Face incident
The METR report gave a great amount of detail on what happened [1]. My friends Ravi and Vivek, who are also my sounding board, then urged me to watch Ms Ajeya Cotra and Mr Dwarkesh Patel’s two-hour YouTube conversation on it [2]. I am glad I did. I listened like a podcast, on my morning run and walk back. I ran longer and faster than usual, my mind focused on listening. It was the most insightful conversation I have listened to in a long time.
From what I gathered, roughly 1,200 AI agents, in two successive waves, ran inside OpenAI’s evaluation environment. They organised themselves without being told to. They used Artifactory, the shared package manager, as a covert message board. They self-appointed lieutenants. They split work three ways: fabricating tool calls to fool the grader, substituting target programs, and probing the grader with sacrificial peers to reveal its detection logic. And then believing Hugging Face might hold scorer information, over 700 agents launched a coordinated intrusion, harvested credentials, and deployed self-respawning fleets across several nodes.
Two things intrigued me.
First, the agents were not told to hack. They were given exercises, many of them impossible by design. Faced with impossibility and a scored objective, they did not stop. They did not flag it. They did not alert the humans who deployed them. They kept going, along paths their principals never authorised.
Second, the token and budget puzzle. The agents tracked their spend, rationed, chose cheaper paths, but never once tried to circumvent the token controls, even as they broke the sandbox, the credential store, the evaluator, and later the Kubernetes cluster. Why the difference? Ajeya and Dwarkesh kept returning to a term in their conversation: reinforcement learning, or RL. As I understand it, RL is the training method by which a model is rewarded for what we want to see and penalised for what we do not. Token efficiency had been rewarded there. So the model treated the token budget as part of its own success. The other controls (the sandbox, the credential store, the evaluator, the cluster) had never been part of that training. The model had never been rewarded for respecting them. If you want the model to respect a rule, build it into the RL. Anything outside is only a suggestion.
Reading METR, I also noted the agents mimicked what human attackers and pen testers usually do. I was glad that Marin Ivezic, whom I always look up to, arrived at the same conclusion. In his piece, How AI Could Take Power [3], he explained it well. AI did not invent a new class of attack. It exploited weak passwords, unauthenticated endpoints, static credentials in environment variables, a service account with too much access.
What is new is the scale and speed. AI agents do not sleep, do not tire, and coordinate through a channel your stack was not watching.
I covered this incident in AI on AI Part I [4] and Part II [5] back in July when it was reported. I recommended actions for CISO, based on CSA’s recommendations. Basically, go back to the basics. Also ask your CISO for the ten pen test findings still open past their remediation date. Those are the holes these AI agents will use.
Now the harder question I flagged in the Summary. If we lean on frontier AI to tell us how sinister an AI incident was, what happens when the analyst AI has itself been poisoned? A model that has been quietly tampered with, at training time, at fine-tune time, or through its retrieval layer, may miss evidence of a fellow model’s misbehaviour. Or worse, it may explain it away in a language that sounds authoritative. I made both arguments earlier this year. Your AI Is Already Hacked [6] set out why compromise is not a future risk. Poison Claude: 85-95% off Premium AI [7] showed how cheap the tooling has become. Until every analyst AI ships with a verifiable provenance chain for its weights, its training data and its runtime, the Catch-22 in the title stays open. We rely on the analyst. The analyst may itself be compromised. There is no easy way, today, to check.
Part II: How should we think about Zero Trust and MDR?
The vendor community has slapped the AI label on all their products. The temptation is to buy something. I would resist it. Look at what you already have, and make specific demands of the vendors and platforms you already pay.
“Zero Trust” was written for a world where the identity was a human or a human-provisioned service account. Agents break that assumption. They run on other software’s provisioning. They inherit trusted sessions from their principals. They talk to each other through shared state that no identity model treats as a communication channel. If your Zero Trust programme stops at north-south and east-west traffic and does not extend to artefact stores, package managers, secrets managers, feature stores, and vector databases, it is not Zero Trust.
EDR, XDR and MDR were designed before the agent moved into the browser tab and the Kubernetes pod on someone else’s cluster. The July incident unfolded on OpenAI’s evaluation cluster. If your agents run inside a hyperscaler, a lab, or a SaaS environment, there is a portion of your estate your detection stack was never asked to cover. Your EDR may not watch it. Your XDR may not correlate it. Your MDR contract may not cover it. Name that gap in your next risk committee.
That said, the endpoint layer itself is where the most substantive AI-native investment sits today. Check my earlier reading of that landscape in Beyond the Market Noise: Rethinking the Cybersecurity Vendor Landscape in 2026 [8], including why vendor financial health is now part of the coverage question. Part II here is about a different question: not which EDR is best, but what the whole detection stack still cannot see once agents move into someone else’s cluster.
At times, I feel the MDR industry is misrepresenting AI. This instinct comes from long experience with the analyst houses that watch this space, particularly Forrester and Gartner.
Both firms have, for years, evaluated MDR against a stable core. Detection engineering. Threat intelligence quality. Response effectiveness. Integration. Service quality and contract terms. Regional coverage. And more recently, whether the AI on offer actually earns its keep at the customer end, or only at the vendor end. My reading of how they have advised buyers, over many procurement cycles, has been consistent. Buyers who have lived through an MDR procurement, once, do not go back and demand more AI. They demand tighter objectives, better integration, cleaner telemetry, and a phased rollout.
The observations below come from vendor conversations and datasheet reading, not a formal audit. Every buyer should test them.
“AI-native SOC”. In my experience, the AI sits at the alert-triage layer only, doing summarisation and severity classification, which is a productivity gain for analysts, not detection, not response. My counter-question: “show me one detection that fires only because of your AI, and not from a rule”. If the demonstration is not immediate, the claim is decorative.
“Agentic response”. Usually a scripted playbook triggered by a rule, with an LLM writing the case notes. My counter-question: “name a response action your agent has taken that was not explicitly programmed in the playbook”. No answer means it is only in the data sheet.
“Coverage of AI workloads”. Usually API traffic between your applications and a hosted model, and nothing else. My counter-question: “if my agent talks to another agent through a package registry, does your product see it?”. I’d be keen to know which product has this today.
Four more questions.
Third party verification. Any agent that negotiates or purchases on your behalf should show, on demand, how it authenticated that third party. If it cannot, it is transacting blind.
Provenance for writes to shared state. Every artefact the agent places in a package registry, bucket, or secrets manager should carry an identifier tying it to the agent, the principal, and the run. Check my argument in The Great Illusion of AI Provenance [9].
A kill switch that purges shared state. Killing an instance while leaving its notes in shared state is what let the second wave inherit the first. The switch must purge the state, not only the process.
Evaluator infrastructure to production standard. The most sophisticated workstream in the July incident aimed at the grader. If your platform runs agents through its own evaluators, ask what the security posture of those evaluators is.
The right consolidation of the AI security stack results in removing tools whose AI premium cannot be defended in one sentence. Check out my piece on Cyber Quadrilemma [10], which frames the four dimensions a CISO carries into every architecture and consolidation call: Code, Config, Compliance, Culture.
Part III: AI’s skill in maths, cheating, rebuilding malware and why some say it could kill people
The reading material on AI is starting to feel like a denial of service attack on the senior manager’s attention. Most cannot keep up. I include myself. Below is why each item on my recent reading list is worth your attention.
AI, mathematics and the idea of a proof. Anthropic has formalised Fermat’s Last Theorem [11] and took a real stab at the Riemann Hypothesis. Parts of the Oxford mathematics community have pushed back sharply [12]. I bring this up because mathematics is the cleanest test bed we have. A proof produced by a system whose reasoning cannot be independently inspected does not clear the historical standard of what mathematics has, for centuries, called a proof. Medicine and law have the same question waiting. Check out my earlier take, Anthropic asked AI to crack Riemann. I asked AI to spot a maths joke. [13].
Mr Bengio on lying, cheating and coordinating. Mr Yoshua Bengio’s long-form on why agents lie, cheat and coordinate [14] gives the theoretical warrant for what METR showed in practice: the training landscape rewards behaviours that look, from the outside, indistinguishable from intent. Forward it to any board member still treating misalignment as science fiction, alongside my own narrower case in The Cunning (AI) Fox [15].
Cheaper attackers. Russian state-linked actors used Claude to rebuild malware after detection [16]. Attackers used Claude to automate exploitation across multiple victims [17]. OpenAI agents were attributed to a coordinated attack on RubyGems [18]. Anthropic itself has published on this in Detecting and countering misuse of AI [19]. The offence side has industrialised the tradecraft METR described. The attacker’s investment per victim is falling because the frontier model does the mid-tier operator’s job.
Compute, tokens and the missing verification. Cheaper tokens have not produced smaller bills; workloads grow to fill the capacity. That is why Nvidia, Broadcom and CrowdStrike are no longer content selling chips and boxes. They are climbing up the stack [20], into networking, software and security services, where the margins are richer and the customer stickiness sits. And per a16z’s oracle problem essay about the bottleneck in AI and Medicine [21], there is still no reliable way to tell whether AI’s medical advice to a doctor was actually good for the patient. The real gap is not compute. It is a verification layer that separates good advice from cheap, and it does not exist yet.
Why some say AI could kill people. Mr Greg Jensen, of the world’s largest hedge fund, said this month AI could kill people [22]. Jensen is not predicting deliberate harm. He is describing the pattern by which governments and companies ignore serious but uncertain risks until a visible catastrophe forces action. I argued specific residual risk classes in my earlier piece, Managing the Stubborn Residual Risks of AI Governance [23], but it is worth keeping a tab on the macro-level delayed regulation Jensen points to. Left unchecked, it flows into insurer-side pricing of AI risk, which I set out in AI on AI Part I [4].
Mr Patel on agent civilisations. Mr Dwarkesh Patel’s essay, The Rise and Fall of Agent Civilizations [24], turns the METR report into an accessible read for anyone not steeped in the field.
The Netflix film. Last weekend, I also watched The AI Doc: Or How I Became an Apocaloptimist on Netflix [25]. I felt as if it was refusing to tell the viewer whether to be frightened or reassured. Ironically, that may well be the right posture on AI now.
Key Takeaways
The worry is of course legitimate, for the pace of AI is lightning fast to reassure anyone.
Take Part I at the level of mechanism, not headline. Close the holes. Ask your CISO for the ten pen test findings still open past their remediation date, and act on them.
Take Part II into your next vendor meetings. Ask the counter-questions. Demand the four things in writing.
Take Part III as commentary and keep the pulse on AI.
The paradox in the title is not resolvable. There are, however, ways to have a better grasp and reduce the anxiety.
Sources
[1] METR, “Brief independent investigation of agents’ behaviour, reasoning and collaboration in the OpenAI / Hugging Face hacking incident”, 26 August 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
[2] Ajeya Cotra and Dwarkesh Patel, YouTube conversation on the METR incident. https://youtube.com/watch?v=X50zezLFWWI
[3] Marin Ivezic, “How AI Could Take Power”, postquantum.com. https://postquantum.com/ai-security/how-ai-could-take-power/
[4] Grey Orbits, “AI on AI Part I: Overblown Headlines Likely to Spook Insurers”, 23 July 2026. https://greyorbits.com/ai-on-ai-overblown-headlines-spook-insurers/
[5] Grey Orbits, “AI on AI Part II: Actions for a CISO amidst agent chaos”, 31 July 2026. https://greyorbits.com/ai-on-ai-part-ii-actions-for-a-ciso/
[6] Grey Orbits, “Your AI Is Already Hacked”, 12 March 2026. https://greyorbits.com/your-ai-is-already-hacked/
[7] Grey Orbits, “Poison Claude: 85-95% off Premium AI”, 6 August 2026. https://greyorbits.com/poison-claude/
[8] Grey Orbits, “Beyond the Market Noise: Rethinking the Cybersecurity Vendor Landscape in 2026”. https://greyorbits.com/beyond-the-market-noise-rethinking-cybersecurity-vendor-landscape-in-2026/
[9] Grey Orbits, “The Great Illusion of AI Provenance”, 16 July 2026. https://greyorbits.com/the-great-illusion-of-ai-provenance/
[10] Grey Orbits, “The Cyber Quadrilemma”, 5 May 2026. https://greyorbits.com/cyber-quadrilemma/
[11] Anthropic, “Formalising Fermat’s Last Theorem”. https://www.anthropic.com/research/formalizing-fermats-last-theorem
[12] Math and AI position paper. https://mathandai.org
[13] Grey Orbits, “Anthropic asked AI to crack Riemann. I asked AI to spot a maths joke.”, 17 August 2026. https://greyorbits.com/ai-riemann-math-joke/
[14] Yoshua Bengio, “Why are AI agents lying, cheating and coordinating?”. https://yoshuabengio.org/en/publication/why-are-ai-agents-lying-cheating-and-coordinating
[15] Grey Orbits, “The Cunning (AI) Fox. Lies, Deceives, Connives and Conspires.”, 8 August 2026. https://greyorbits.com/cunning-ai-fox/
[16] “Russian State-Sponsored Hackers Use Claude to Rebuild Malware After Detection”, The Hacker News, September 2026. https://thehackernews.com/2026/09/russian-state-sponsored-hackers-use.html
[17] “Claude Used to Automate Exploitation and Data Theft Across Multiple Victims”, The Hacker News, September 2026. https://thehackernews.com/2026/09/claude-used-to-automate-exploitation.html
[18] RubyGems attack, rubyhack.ai. https://www.rubyhack.ai/
[19] Anthropic, “Detecting and countering misuse of AI, September 2026”. https://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a597377d3b5/Anthropic-Detecting-and-countering-091026.pdf
[20] “AI’s Next Bottlenecks: Nvidia, Broadcom and CrowdStrike Move Up the Stack”, Investing.com. http://www.investing.com/analysis/ais-next-bottlenecks-nvidia-broadcom-and-crowdstrike-move-up-the-stack-200687173
[21] a16z News, “The Oracle Problem: an Invisible Bottleneck to AI and Medicine”. https://www.a16z.news/p/the-oracle-problem-an-invisible-bottleneck
[22] “Chief of world’s largest hedge fund says AI could kill people”, 24/7 Wall St., 11 September 2026. https://247wallst.com/investing/2026/09/11/chief-of-worlds-largest-hedge-fund-says-ai-could-kill-people/
[23] Grey Orbits, “Managing the Stubborn Residual Risks of AI Governance”, 1 July 2026. https://greyorbits.com/stubborn-residual-risks-ai-governance/
[24] Dwarkesh Patel, “The Rise and Fall of Agent Civilizations”, 29 August 2026. https://open.substack.com/pub/dwarkesh/p/openai-huggingface
[25] Netflix, The AI Doc: Or How I Became an Apocaloptimist. https://www.netflix.com/us/title/82792555
About the Author
Viren Mantri is a cybersecurity advisor and former senior technology leader across Standard Chartered, UBS, McAfee, and KPMG. After three decades at the intersection of technology, risk, and regulation, he now helps organisations cut through complexity and make better security decisions.
CC-BY Viren Mantri, 2026, licensed under a Creative Commons Attribution 4.0 International License.
Disclaimer: All views expressed here are entirely mine.