Scientists invented a fake disease. AI told people it was real

"Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice."

"The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, major artificial-intelligence systems began repeating the invented condition as if it were real."

"Even more troublingly, other researchers say, the fake papers were then cited in peer-reviewed literature. Osmanovic Thunström says this suggests that some researchers are relying on AI-generated references without reading the underlying papers."

Bixonimania, a non-existent medical condition, was created by Almira Osmanovic Thunström to test the reliability of large language models. After uploading fake studies, major AI systems began treating it as real. The experiment revealed that some researchers cited these fabricated papers in peer-reviewed literature, indicating a reliance on AI-generated references without proper verification. The condition was first mentioned in blog posts and later appeared in preprints attributed to a fictional author, raising concerns about the integrity of academic research in the age of AI.

Read at Nature