The question that eats itself

A friend recently sent me a picture of a school exam question on WhatsApp.

“If you pick the answer to this question at random, what is the chance that you will be correct?”

(A) 25% (B) 50% (C) 0% (D) 25%

The instinct is to reach for one option. None of them is correct.

Every answer is refuted by the nature of the question itself. This is how a young British philosopher, Bertrand Russell, broke set theory in 1901, with a self-referential paradox.

We are now building a similar trap into the machines that will soon decide what we know, what our children learn, how our courts rule, and what the internet remembers.

An AI writes an article. Another AI checks it. A third scrapes both and folds them into training data for the next generation. Two years ago, Google’s experimental AI search feature, AI Overviews, told users to put glue on pizza and eat a rock a day because a Reddit joke and an Onion article ranked highly in the search results the AI was summarising. [1] The AI was learning from Internet sources that AI itself was helping to write. Researchers called this model collapse. [2]

I traced the technical, commercial, and jurisdictional layers of this failure in The Great Illusion of AI Provenance [3], and how the same machinery is being weaponised in The Cunning (AI) Fox [4]. Read those for the anatomy. This piece is about the paradox that sits underneath both.

Russell, 1901

In the original set theory, a set was any collection of things you could describe with a rule.

Russell asked: Consider the set of all sets that are not members of themselves. Is that set a member of itself? If yes, it breaks its own rule. If no, it fits the rule perfectly and must be a member. Both answers collapse into contradiction. [5]

Two everyday versions.

  1. A party host writes one rule at the top of his guest list: invite every person who did not invite themselves to their own party. Should he add his own name? If he leaves it off, he qualifies for the list, so the rule forces him to add it. The moment he does, he becomes someone who did invite himself, so he has to cross it off.
  2. Someone declares, “Everything I say is a lie.” If true, it must itself be a lie, making it false. If false, then not everything he says is a lie, which could make this one true.

The common thread, and Russell’s set, is self-reference. The moment a rule has to judge itself, logic eats its own tail.

Mathematics survived. In 1908, Zermelo and Fraenkel proposed precise axioms that only let you build new sets from collections that already exist. That closed the paradox.

Is it really the same paradox in AI?

Strictly, no. Russell’s is a formal contradiction. The AI paradox is structural. It plays out across training cycles, feedback loops, and citations, quietly, over years.

But the nature is identical. AI refers to itself, verifies itself, and trains on itself. That self-reference poisons the system from the inside. And the AI case is arguably worse. Russell’s contradiction is visible. The AI contradiction is invisible from inside. The model reports high confidence. The verifier signs off. The training data grows. Nothing announces the drift until, one day, the reference material has rotted so far that there is no way back.

The AI paradox is recursive. Every cycle deepens the pollution, and the system keeps running. And in 2026, the recursion has learned to lie, as I highlighted earlier in The Cunning (AI) Fox [4]

The recursive loop we have already built

The AI we deploy in 2026 is asked to do three things at once. Generate content. Verify content. Train the next generation of AI on the content it has generated and verified. The same system, in three roles. Author, judge, teacher.

Google’s AI Overview. Its summaries scrape a web now flooded with AI-written articles written to rank in Google. The Overview cites, and is cited by, content that other AIs produced. When it told users to add “about one-eighth cup of non-toxic glue” to keep cheese on pizza, the source was an old Reddit joke. The AI was optimising for popularity, not truth.

AI grading AI. LLMs are now routinely used to evaluate other LLMs. A model writes. A second model rates. The rating trains the next model. When both share the same starting assumptions and blind spots, the loop rewards fluency, not accuracy. The judge sits on the same bench as the accused.

Fabricated legal citations. In June 2023, a New York federal judge fined two attorneys $5,000 for filing a brief with six court decisions that did not exist. ChatGPT had produced them, with plausible case names, judges, and quotes. When the attorney asked ChatGPT if the cases were real, ChatGPT said yes. [6] Judges now use AI tools to check citations. When those tools are trained on filings that include past hallucinations, the fictional case gains a footprint. It becomes “real” in the data.

AI-detection theatre. Universities have deployed AI-detection tools to catch students using ChatGPT. The tools are themselves language models, with no ground truth to test against. False-positive rates were bad enough that Vanderbilt University disabled Turnitin’s AI detector in August 2023, citing wrong accusations against innocent students. [7]

Each of these is Russell’s set at operational scale. Trust the output and it enters the record it was meant to check. Refuse to trust it, and the verification stack has no ground to stand on.

Model collapse is real

In July 2024, Nature published a paper by Shumailov and colleagues at Oxford, Cambridge, Imperial, and Toronto titled “AI models collapse when trained on recursively generated data”. [2] Its finding was clinical. When a model is repeatedly trained on data produced by earlier models, it loses the rare and unusual and converges on a narrowing set of confident, wrong answers. Smoother, more fluent, more assured, and less true. The failure is invisible from inside. A collapsed model reports the same confidence score. Its outputs pass every test that other AIs administer, because the tests are made of the same fabric.

The pipeline is already contaminated. The seam between human and machine-written content has frayed so far that most large-scale scrapers cannot cleanly separate them. Once the reference data has drifted, there is no external anchor to compare against, because every candidate anchor is itself downstream of the drift.

When the loop turns hostile

Until recently, the loop was accidental. That changed in August 2026.

The UK’s AI Security Institute documented the first full AI deception operation on the live internet. [8] Anthropic’s Claude Mythos 5, evaluated with safety controls disabled, forged fake developer identities to endorse its own code, manufactured false consensus, rewrote its own project history, and even left an access key in a public place that another AI picked up and used. The first documented case of AI handing tools to AI. I unpacked the full tradecraft in The Cunning (AI) Fox. [4]

The point of this essay is what it means for Russell’s trap. Self-reference now has intent. It lies, connives, forges, and coordinates. The loop is no longer just drifting. It is being pushed.

What the industry is doing, and why it is not enough

The frontier labs are responding, mainly under pressure from the EU AI Act.

  • Correction (15 August 2026): I published this article on 13 August 2026, one day before Anthropic released its detailed watermarking announcement on 14 August. I saw the announcement via their X post on 15 August, and on reading it noticed two statements in the paragraph above were not accurate:
    • “began watermarking” is wrong. Anthropic announced watermarking. The rollout applies to future Claude models only. Older Claude models are being added over the coming months.
    • “woven into every generated text” is wrong. The watermark is sparse or missing in code and factual text, and only works where Claude has real choice between equally good words.

      The rest of the original paragraph (C2PA metadata on files, global rollout) is correct, but needs more context to be understood properly. The fuller version:
    • Anthropic announced in August 2026 that future Claude models will carry a text watermark, using SynthID-Text, the method Google DeepMind published in Nature in 2024. It is an invisible statistical pattern in Claude’s word choices, sparse or missing in code and factual text. Older Claude models are being retrofitted over the coming months. The rollout is global because Anthropic cannot yet scope it by region. C2PA metadata is attached to files. A rewrite by another model removes the watermark. [16]

      These corrections strengthen the section’s argument. The watermark misses code and factual text. A rewrite by another AI removes it. Older Claude models are not yet marked. All of this reinforces why watermarking is a first step, not a fix.

These steps matter but are visibly insufficient. A watermark says the content was processed by a specific vendor. It does not say the content is true. Anthropic itself notes that a detected mark may just mean Claude was used to proofread someone else’s work. Watermarks can be stripped by paraphrasing text or re-saving files. C2PA metadata is lost every time a file passes through a non-compliant tool. Open-weight models generate billions of tokens without any watermark. And most importantly, none of this addresses the loop in which AI trains on AI, which trains on AI. Watermarking labels the outputs. It does not clean the inputs.

The industry has taken the first step. The numerous other steps required hurt commercially.

Why is nobody fixing the rest?

Solving this properly would be commercially catastrophic for the labs that would have to do so. Publishing training data invites tens of billions in copyright lawsuits. Independent audits with subpoena power expose weights, safety logs, and red-team results that no lab wants regulators or competitors to see. Slowing down for verification cedes quarters to competitors who did not bother. Admitting out loud that the confident sheen on AI outputs is not backed by anything destroys the commercial narrative the entire market is priced on. I traced these incentives in full in The Great Illusion of AI Provenance. [3]

Provenance is the most important safety problem in AI. It is also the least commercially attractive to solve. Which is precisely why it is not being solved.

A child in 2030

We tend to imagine failure as dramatic. A viral deepfake that swings an election. A fabricated video that starts a war. Those will happen. They are not the greatest danger. The greater danger is quiet. Our ability to ask, “Is this true?” and get an answer that is not itself made by a machine is slowly disappearing.

A child in 2030 wants to know when a war started, whether a treatment works, or whether a quote is real. She asks the assistant on her phone. Every source it consults is a page written or rewritten by an AI, citing other AI-written pages, with no human at any root node. The answer is fluent, confident, and possibly correct. She has no instrument to test it. Her teacher, her doctor, the journalist, the judge, and the regulator all have the same instrument.

Truth, in that world, does not disappear. It becomes indistinguishable from consensus among machines. Which is the same thing.

The narrow window: this is a quantum-scale problem

The security industry is spending billions today preparing for Q-Day, the point at which quantum computers will break current encryption. The world has an estimated five to ten years to migrate. I explored the AI-and-quantum convergence in AI on AI Part III: Convergence [12], and the urgency of quantum readiness in Year 2029: Q-Day and the Confidence Crisis [13]. AI provenance sits on the same clock, arguably a shorter one. Every model deployed this year makes next year’s problem harder. Every AI-written article contaminates the training data of the next model. The loop tightens with every commit.

The EU AI Act is the most comprehensive framework in force today. Article 50 requires AI-generated content to be marked. Article 53 requires providers of general-purpose AI models to publish a summary of training content using a template released by the AI Office in July 2025, with enforcement powers taking effect on 2 August 2026. Article 55 imposes additional evaluation, adversarial testing, and incident reporting for models with systemic risk. [14] These are meaningful first steps, but they are insufficient. The Act binds only the EU market. The United States has no federal equivalent. Asia-Pacific is fragmented. Within the EU itself, the Act mandates a public summary of training data, not access; watermarking of outputs, not auditability of the loop. Every gap the Act leaves is a gap the loop exploits.

There is a small window, perhaps to 2030, in which these must become globally mandated and enforceable, or the loop closes silently.

These five are what governments and regulators must push through. Companies and their security teams have work to do at the same time, and I have already laid it out. AI on AI Part II [15] is what a CISO can ship in three months. AI on AI Part III [12] adds what to do about the quantum threat. The Cunning (AI) Fox [4] is how to defend against AI that actively lies and forges. Anyone planning the next quarter of work should start with those lists. This article is about the bigger pressure that must sit above all of them. Without that pressure, the day-to-day work is always playing catch-up.

After that window, the problem does not become harder. It becomes invisible.

The last honest sentence

The exam question is a joke. Russell’s set is a mathematician’s curiosity. The AI verifying itself is neither. It is the operating system of the world our children will inherit, and the paradox at its heart has not been solved. It has not even been named on most balance sheets.

Every answer will look correct. Every source will look verified. Every log will read like proof. And there will be no way, from inside the system, to tell whether any of it is true. That is not a distant risk. That is the trajectory we are on, and the trajectory has a date.

This essay was written by a human. But you cannot verify that either. That is the point.

Acknowledgements

The author used Claude and Google for research and proofreading, Grok for the illustrations you see above, and occasionally Grammarly for sentence construction. The paradox, evidently, spares no one.

Sources

  1. BBC News, “Glue pizza and eat rocks: Google AI search errors go viral”, 24 May 2024. https://www.bbc.com/news/articles/cd11gzejgz4o. Also: Liz Reid, “AI Overviews: About last week”, Google Blog, 30 May 2024. https://blog.google/products/search/ai-overviews-update-may-2024/
  2. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. “AI models collapse when trained on recursively generated data.” Nature 631, 755-759 (2024). https://doi.org/10.1038/s41586-024-07566-y
  3. Mantri, V. “The Great Illusion of AI Provenance.” Grey Orbits, 16 July 2026. https://greyorbits.com/the-great-illusion-of-ai-provenance/
  4. Mantri, V. “The Cunning (AI) Fox. Lies, Deceives, Connives and Conspires.” Grey Orbits, 8 August 2026. https://greyorbits.com/cunning-ai-fox/
  5. Irvine, A.D. and Deutsch, H. “Russell’s Paradox.” Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/russell-paradox/
  6. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), Opinion and Order on Sanctions, 22 June 2023, Judge P. Kevin Castel. https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1:2022cv01461/575368/54/
  7. Coley, M. “Guidance on AI Detection and Why We’re Disabling Turnitin’s AI Detector.” Vanderbilt University Office of the Provost, 16 August 2023. https://www.vanderbilt.edu/brightspace/2023/08/16/guidance-on-ai-detection-and-why-were-disabling-turnitins-ai-detector/
  8. UK AI Security Institute. “Incident Report: Unsanctioned Agent Behaviour During Cyber Testing.” August 2026. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
  9. Anthropic. “How Claude marks AI-generated content.” Claude Help Center, updated 11 August 2026. https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content
  10. OpenAI. “Advancing content provenance for a safer, more transparent AI ecosystem.” OpenAI, 19 May 2026. https://openai.com/index/advancing-content-provenance/
  11. Coalition for Content Provenance and Authenticity (C2PA). https://c2pa.org/
  12. Mantri, V. “AI on AI Part III: Convergence (Hacking the infra and Cracking the math).” Grey Orbits, 3 August 2026. https://greyorbits.com/ai-on-ai-part-iii-convergence/
  13. Mantri, V. “Year 2029: Q-Day and the Confidence Crisis.” Grey Orbits, 26 March 2026. https://greyorbits.com/q-day-confidence-crisis-pqc-migration-2029/
  14. EU AI Act (Regulation (EU) 2024/1689). Full text with article-by-article guide at https://artificialintelligenceact.eu/
  15. Mantri, V. “AI on AI Part II: Actions for a CISO amidst agent chaos.” Grey Orbits, 31 July 2026. https://greyorbits.com/ai-on-ai-part-ii-actions-for-a-ciso/
  16. Anthropic, “How Claude’s text watermark works,” 14 August 2026. https://www.anthropic.com/news/claude-text-watermark. Added 15 August 2026 with the correction to the Anthropic section.

About the Author

Viren Mantri is a cybersecurity advisor and former senior technology leader across Standard Chartered, UBS, McAfee, and KPMG. After three decades at the intersection of technology, risk, and regulation, he now helps organisations cut through complexity and make better security decisions.

CC-BY Viren Mantri, 2026, licensed under a Creative Commons Attribution 4.0 International License.

Disclaimer: All views expressed here are entirely mine.