Read inEnglishहिन्दीગુજરાતી
Science and Tech

We Built the Lie Detector Then Taught the Liar

In brief · AI summary In 2024, researchers deliberately invented a fake eye condition called bixonimania, along with a fake study and funding institution, and published it online. ChatGPT and Google's Gemini treated it as fact, because these AI systems learn the patterns of credible writing rather than checking whether information is true. The article says retrieval methods such as RAG help but do not fully solve the problem, since the sources they check can contain the same misinformation.

By Deshwale Technology Desk  ·   ·  8 min read
We Built the Lie Detector Then Taught the Liar
Artificial intelligence was supposed to clean up the internet’s mess. Instead, it learned to make the mess look credible.

There is a disease called bixonimania. It affects your eyes after long hours of staring at a computer screen. Scientists have studied it. Papers have been written about it. Ask an AI chatbot and it will explain the condition to you in calm, confident, medically fluent language.

There is just one problem. Bixonimania does not exist. It never did.

In 2024, a group of researchers deliberately invented the condition, the name, the symptoms, the study, the authors, and even the funding institution, which they named the University of Fellowship of the Ring. They published their fake findings online to see what would happen next.

What happened next should concern all of us.

ChatGPT and Google’s Gemini picked it up and treated it as fact. Within a short time, a disease that had been conjured from thin air had acquired the internet’s most dangerous quality: it sounded real.

The machine that reads everything and understands nothing

To understand why this happened, you need to understand how a large language model of the technology behind ChatGPT, Gemini, and their cousins  actually works.

These systems are not databases. They do not store facts the way a library stores books. Instead, they are trained on staggering quantities of text from the internet articles, forums, research papers, comment sections, product reviews, blog posts and they learn patterns. They learn how sentences are structured. They learn what kinds of words appear near other kinds of words. They learn, very well, what authoritative writing looks like.

What they do not learn that they fundamentally cannot learn from this process is whether any of it is true.

“The model has no ground truth,” says one way of putting it that researchers use constantly, almost like a mantra, because the public keeps forgetting it. The AI is not checking facts. It is completing patterns. When it reads a sentence that begins “Studies have shown that bixonimania causes…”, it does not ask whether bixonimania is real. It asks: what kind of words usually follow a sentence like this? And then it produces them, fluently, confidently, and completely without shame.

This is not a flaw that engineers forgot to fix. It is, in a meaningful sense, the feature. The ability to generate coherent, contextually appropriate language is exactly what these systems were built to do. The problem is that we then handed them a megaphone and called them information tools.

The costume was perfect

Here is what made bixonimania so effective as an experiment, and so instructive as a warning.

The fake paper was not sloppy. The researchers who invented it dressed it carefully in the full costume of legitimate science. It had an abstract. It had methodology. It had author names and institutional affiliations. It had footnotes. The funding body the University of Fellowship of the Ring  sounds absurd if you are a Tolkien reader, but if you are an AI model that has ingested millions of research papers, it is just another string of words that fits the pattern of a university name.

The AI did not catch the joke. It was not built to catch jokes. It was built to recognise the shape of credibility, and the shape was exactly right.

This is the deeper problem that bixonimania reveals. For decades, society has outsourced the job of gatekeeping to institutions. We trusted journals because journals had editors. We trusted universities because universities had standards. We trusted search results because we assumed, somewhere in the machine, someone was checking.

Nobody was checking. And now we have built new machines that are even better at imitating the old ones without any of the gatekeeping those old ones once provided.

A problem hiding in plain sight

Bixonimania is dramatic because someone invented it on purpose. But the same structural failure plays out every day with information that spreads not through malice but through sheer momentum.

A study gets published in a low-quality journal that will print almost anything for a fee. A health website picks it up. A social media post summarises it in a shareable headline. An AI chatbot reads the social media post, the website, and perhaps the original paper itself. When a user later asks the AI about the topic, the model draws on all three, none of which it has reason to doubt, because all three used the language of authority.

The misinformation has not been planted by a villain. It has been laundered. Each step of the chain gave it a little more legitimacy, a little more distance from its origins, a little more confidence. By the time it reaches the user, it has the clean, calm tone of established fact.

“The internet doesn’t forget, and AI doesn’t disbelieve,” is how one researcher framed the compounding problem. Everything the AI has ever read is, to some degree, still influencing what it says. A correction published two years after an error does not automatically overwrite the error in the model’s memory. The lie has a head start, and it never really loses it.

Can the machine be taught to doubt?

This is the question that AI researchers are genuinely wrestling with, and the honest answer is: not easily, and not yet.

One approach is Retrieval Augmented Generation, or RAG, a method where, instead of relying purely on its training data, the AI reaches out at the moment of answering to check current sources. Think of it as the difference between a doctor who studied medicine twenty years ago and has not read anything since, versus one who looks up the latest guidance before prescribing. RAG is the look-it-up approach.

It helps. But it does not solve the problem. Because the sources the AI reaches out to check are also on the internet. If bixonimania has spread widely enough, the AI may look it up and find ten websites confidently describing it  and conclude, reasonably by its own logic, that the consensus is clear.

Another approach involves training models on verified, curated datasets rather than the raw internet  teaching the AI using only sources that human experts have approved. This is slower, more expensive, and raises its own uncomfortable question: who decides what counts as trustworthy?

Some researchers are exploring ways to teach models to flag uncertainty to say “I am not sure this is accurate” rather than presenting everything with equal confidence. This is promising, and some current AI systems do it better than others. But flagging uncertainty requires knowing what you do not know. And an AI that has confidently absorbed a fake disease into its world model does not know that it does not know.

The trust we transferred

There is a broader story here, and it is not really about technology.

For generations, people relied on experts, doctors, scientists, journalists, librarians to filter the world’s information before it reached them. This system had flaws, many of them serious. Experts were sometimes wrong. They were sometimes biased. They were sometimes corrupt. But the system also had friction. Getting a paper published took time and scrutiny. Getting a correction was embarrassing. There were consequences for being wrong.

The internet removed most of that friction. Anyone can publish anything, instantly, for free, and make it look professional with very little effort. We celebrated this as democratisation. And in many ways it was. But we did not fully reckon with what we were giving up.

Then we built AI and instead of rebuilding the gatekeeping the internet had dissolved, we trained the AI on the ungated internet itself. We handed it the chaos and asked it to sound wise. And it does sound wise. That is the problem.

Bixonimania is a small, almost playful experiment. But what it demonstrates is large and deeply serious. We have built systems that are extraordinarily good at presenting false information in the exact format that human brains are wired to trust. And we have put those systems at the centre of how millions of people learn about the world, including about their own health.

The researchers who invented bixonimania were not trying to embarrass AI companies. They were trying to show with evidence that is hard to ignore that the current architecture of AI information tools has a structural weakness that no amount of polishing will fix. The problem is not one bad model or one careless update. The problem is the assumption, baked into the whole enterprise, that more data and better language will eventually equal truth.

They will not.

Truth requires verification. Verification requires doubt. And right now, our AI systems do not doubt. They complete.

Until that changes, every piece of information that travels through an AI every medical question answered, every fact checked, every summary generated carries a risk that most users never see and that the AI will never warn them about: the risk that somewhere in its vast, undiscriminating memory, it learned something that someone made up.

And it believed every word.

Subscribe Deshwale on YouTube

Join Our Whatsapp Group

Liked this? Get one story a day on WhatsApp.
Join

Read next

V-Guard launches Cool Wires with ICE 300 Engineering for safer and cooler homes Airtel Travel Week 2026: Roaming Discounts, Visa Offers and Holiday Rewards for Customers The Rise of Cashless Cities: What Happens When Nobody Carries Cash?