Why We Are Pretending to Know What We Cannot Verify

A long-form essay on the machinery of verification that does not exist, the stories we tell ourselves, and why we are all flying blind together.

The photograph that broke the machine

In March 2026, missiles destroyed a girls’ primary school in Minab, southern Iran. Days later, an aerial photograph of the town’s cemetery began circulating. Dozens of freshly dug graves, arranged in orderly rows. It became one of the most direct visual records of civilian casualties in the war.

Then the AI fact-checkers weighed in.

Google’s Gemini confidently asserted that the image was from Kahramanmaraş, Turkey. Mass burials, it said, from the 2023 earthquake. X’s Grok, equally confident, placed it in Jakarta. A COVID-19 mass burial from 2021. Both cited specific dates, locations, and original sources. Both were fabricated.

When The Guardian pushed back, Gemini revised its finding. Then revised it again. Then again. Every correction came with the same unwavering tone. Every correction was wrong.

The photograph was authentic. Researchers matched it to satellite imagery, video footage, and drone shots. No signs of tampering. [1]

This is not a story about AI getting things wrong. It is a story about the impossibility of provenance as currently imagined. Gemini and Grok did not simply hallucinate. They invented provenance. They fabricated chains of custody with the same statistical confidence they bring to everything else. That is what they are optimised to do. Produce plausible-sounding answers, not verifiable ones.

Creative writing about data (the technical layer)

Every major AI company now points to provenance as a saving grace. We track our data. We log our training. We can show you the chain of custody. It sounds reassuring. It sounds like science.

Consider what real provenance would require.

That is not a log file. It is a parallel universe of metadata, larger than the model itself.

Now consider what we actually get.

In computer science, data provenance records the entities and processes behind a resource. The W3C’s PROV standard models this as a graph of entities, activities, and relationships. [2] That is the theory.

In practice, the most prominent industry effort is C2PA, founded in 2021 by Adobe, Arm, BBC, Intel, Microsoft, and Truepic. [3] It embeds a cryptographically signed manifest inside a media file, recording who made it, when, and whether AI was involved.

Three problems are already visible.

Even flagship implementations have stumbled. Nikon added C2PA to the Z6 III via a firmware update in August 2025. Within days, an independent researcher demonstrated how to get the camera to sign an AI-generated image. Nikon suspended the service in September and revoked all certificates issued during the affected period. [4]

So what does provenance mean in practice? It means a summary. A human-readable paragraph that sounds like provenance but is really creative writing about data. It tells a story. It does not provide receipts.

Opacity by design (the business layer)

Even if the technology worked perfectly, there is still a deeper problem.

Provenance verification requires access. From the training data to the model weights to the full pipeline. Commercial AI is increasingly a black box.

This is not an engineering oversight. It is a business model. Companies that spent billions training models have no incentive to expose their internals to independent auditors. Proprietary secrets is the polite term. Opacity is the product.

Consider what verification would actually require.

Even if the technology existed, we would not want it. Real provenance would expose copyrighted images, biased annotations, and sketchy web scrapes. It would reveal that “verified sources” often means Reddit and fanfiction. It would force companies to admit what they cannot say out loud. No one, including them, fully understands their own models.

So we accept the illusion. We take the summary. A feel-good label such as “trained on diverse, high-quality data.” We have traded verifiability for convenience and are calling it progress.

Whose AI is this? (the jurisdictional layer)

Not everyone has given up.

My former colleague Iolaire McKinnon has built the most rigorous attempt I have seen to make AI jurisdiction verifiable in practice. His tool, Borderlint, maps which laws actually apply to an AI system across residency, sovereignty, and provenance. [5] It shows that a single AI model can fall under three different jurisdictions at once, depending on which lens you apply.

Take one of McKinnon’s own examples. DeepSeek-R1 served via AWS Bedrock in Hong Kong. Where is it from?

Three different answers to one question, all correct. This is not a theoretical edge case. It is the new normal.

The Borderlint knowledge base [6] now covers 92 AI providers and 71 model developers, mapping jurisdictions and legal regimes (PDPO, PIPL, GDPR, APPI, PIPA, PDPA, the Privacy Act, and others).

Even so, Borderlint runs into an inherent limit. It answers who owns the weights, whose law applies, and where the bytes rest. It does not open the black box. It cannot tell you which images, texts, or annotations were used to train the model. It cannot verify consent or licensing for the data inside. Jurisdictional layer, yes. Training-data layer, no.

Even if you could track every byte of training data, you would still not have a single, objective origin for an AI model. Provenance is not just technically difficult. It is legally and politically contested.

The web of truth

The industry has an answer of its own.

Dario Amodei, CEO of Anthropic, has expressed optimism that AI can bridge data gaps because, in his words, “all the true things are connected in the world, whereas lies are kind of disconnected.” [7]

True facts form a coherent web. Missing truths can be inferred. It is a seductive idea, and it may hold for descriptive facts. It does not solve provenance. Provenance can tell you where data came from. It cannot tell you whether the values encoded in that data are correct, or compatible, or just. Those are questions for humans.

Nobody home

We are outsourcing more than fact-checking. We are outsourcing how we know things.

Elizabeth Fricker, of Oxford’s Faculty of Philosophy, argues that AI “testimony” is fake. It mimics human assertion but lacks understanding or responsibility. [8]

When a human testifies, they take responsibility. They can be corrected. They can be exposed. Their biases can be interrogated.

When an AI testifies, there is nobody home. No one to hold responsible. Just a statistical pattern in the shape of an assertion. Yet the output is formulated to look exactly like human communication. Same cues of authority, confidence, and coherence. We respond to it the way we respond to a person, even when we know it is not one.

Today a child learns: read the book, check the source, ask the expert. That child’s child may learn: ask the AI, check its confidence score, trust the log. When the log is a self-referential story, an AI citing an AI citing an AI, the next generation will have no tools left to question it.

That is not the AI’s fault. It is ours.

The AI eating its tail

The greatest danger is often overlooked.

AI outputs, with their fabricated sources and confident errors, are already flowing back into the training data of future models. When a false AI summary is shared, discussed, and cited, it becomes part of the internet. When a future AI scrapes that internet, it ingests its predecessor’s hallucinations as fact.

The failure mode is no longer wrong answers. It is wrong citations. Plausible-looking references that lead nowhere. Invented sources that sound authoritative. Evidence trails that terminate in empty space.

A study in Scientific Reports evaluated ChatGPT’s ability to simplify scientific abstracts. [9] On a ten-point scale, “hallucination presence” scored 6.01. “Technical term usage” scored 6.95. Both fell below the study’s own threshold of 7 for acceptable performance. Even when the summaries appeared clear and readable, they smuggled fabricated content past readers who were paying attention.

Bias masqueraded as fact

Provenance struggles with bias. AI does not care about race or nationality. We do. When we feed politicised, historically contingent categories into a system, its provenance log becomes a receipt for our prejudices.

Imagine a border-screening AI trained on passport applications from 50 countries and facial recognition data from surveillance feeds. When a Syrian-born doctor with a German passport applies, the tool flags a “provenance mismatch.” The system was never taught that citizenship is a legal construct, not a genetic fact. But its provenance record says it “used official data”, and so the bias arrives wearing a suit and calling itself objective.

At scale, this turns historical categories into seemingly neutral outputs and masks omissions behind the polish of impartiality.

Not a reason to abandon

None of the above is an argument against trying. If anything, it demands more effort, not less.

Researchers are developing promising approaches. PROLIT, from Roma Tre and the University of Birmingham, focuses on the data-preparation stage of machine learning. [10] It uses a language model to rewrite user pipelines into a trackable form, segment the code so provenance can be associated with each snippet, and generate natural-language narratives of what each operation did. Not a complete solution. A serious effort to make one stage honest.

The Auditable Autonomous Research (AAR) standard, proposed in a 2026 arXiv paper by Rasheed and colleagues, argues for claim-level auditability. [11] Not just tracing what the system did, but encoding which sources substantiate each claim and how. A shift from action logs to semantic provenance.

These efforts push towards building trust rather than proving it. Towards systems that can be held accountable, not systems that merely look accountable. They can flag inconsistencies. They cannot resolve genuine conflicts of value. They can make opacity visible. They cannot make it disappear.

Radical honesty

If provenance in its current form is an illusion, what should we do?

Flying blind

There is only trust, dressed in jargon. And that trust currently sits with for-profit corporations who have every incentive to make their logs look good, not be good.

We are not heading towards a world of verified truth. We are heading towards a world of verified-sounding fictions. That is the risk worth naming.

The machinery to verify was never fully built. It may be too late to build it now. This is not a counsel of despair. It is a call for radical honesty. Build better instruments. Not because they will give us perfect vision, but because they will at least tell us when we are about to hit something.

Preserve the messy, contradictory, human archives that anchor us. They are biased. They are incomplete. But they are ours. And they are the only ground truth we have left.

So the question is this. Are we brave enough to admit we are flying blind? Or will we keep pretending that a JSON file called “provenance” is the same as knowing where we came from?

This essay was written by a human. But you cannot verify that either.

That is the point.

Sources

  1. The Guardian. “A photo of Iran’s bombed schoolgirl graveyard went viral. Why did AI say it wasn’t real?” 17 March 2026. https://www.theguardian.com/global-development/2026/mar/17/atrocity-ai-slop-verify-facts-iran-minab-graves
  2. W3C. PROV Overview. Standard for data provenance on the web. https://www.w3.org/TR/prov-overview/
  3. Coalition for Content Provenance and Authenticity. Technical specifications and adoption notes. https://c2pa.org/
  4. PetaPixel. “Nikon Suspends C2PA Functionality on the Z6 III Due to Authentication Issue.” 5 September 2025. https://petapixel.com/2025/09/05/nikon-suspends-c2pa-functionality-on-the-z6-iii-due-to-authentication-issue/
  5. Borderlint tool (Iolaire McKinnon). https://github.com/iolairus/borderlint
  6. Borderlint knowledge base https://iolairus.github.io/borderlint/
  7. Queloz, M. “Can AI Rely on the Systematicity of Truth? The Challenge of Modelling Normative Domains.” Philosophy & Technology, 2025. https://link.springer.com/article/10.1007/s13347-025-00864-x
  8. Elizabeth Fricker. “On the metaphysical and epistemic contrasts between human and AI testimony.” Inquiry, vol. 69, issue 6 (2026), pp. 2859–2884. Published online 29 September 2025. https://www.tandfonline.com/doi/full/10.1080/0020174X.2025.2553297
  9. Dogru-Huzmeli, E. et al. “Evaluating ChatGPT’s ability to simplify scientific abstracts for clinicians and the public.” Scientific Reports 15, 33466 (2025). https://www.nature.com/articles/s41598-025-11086-8
  10. Lazzaro, P.L., Lazzaro, M., Missier, P. and Torlone, R. “PROLIT: Supporting the Transparency of Data Preparation Pipelines through Narratives over Data Provenance.” EDBT 2025 Demo Track. https://openproceedings.org/2025/conf/edbt/paper-336.pdf
  11. Rasheed, R.A., Banerjee, S., Mukherjee, A. and Hazra, R. “From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agents.” arXiv:2602.13855, 2026. https://arxiv.org/abs/2602.13855

About the author

Viren Mantri is a cybersecurity advisor and former senior technology leader across Standard Chartered, UBS, McAfee, and KPMG. After three decades at the intersection of technology, risk, and regulation, he now helps organisations cut through complexity and make better security decisions.

CC-BY Viren Mantri, 2026, licensed under a Creative Commons Attribution 4.0 International License.

Disclaimer: All views expressed here are entirely mine.