Some signs of AI model collapse begin to reveal themselves

Are you noticing AI search results aren't as sharp as they used to be? If you're finding it harder to get reliable data, especially financial figures, from AI tools like market share statistics from 10-K reports, you're not alone.

This isn't just a glitch; it's a sign of "AI model collapse" starting to appear. This phenomenon occurs when AI systems are trained on content generated by other AIs. Think of it as GIGO - Garbage In, Garbage Out - but on a massive scale. Errors and biases from earlier AI outputs get baked into new models, compounding over generations. This leads to AIs that are less accurate, less diverse, and less reliable, becoming, as one Nature paper put it, "The model becomes poisoned with its own projection of reality."

The issue stems from three factors: error accumulation across generations, the loss of "tail data" (rare or unique information gets smoothed out, blurring concepts), and feedback loops that reinforce narrow, repetitive patterns.

Even advanced techniques like Retrieval-Augmented Generation (RAG), which lets AIs pull from external sources like databases or documents, aren't a silver bullet. A recent Bloomberg study found that while RAG can reduce hallucinations, it can also increase the risk of sensitive data leaks, misleading analyses, and biased advice when dealing with sensitive financial data.

Why is this happening? A major driver is the sheer volume of AI-generated content flooding the internet. With companies like OpenAI reportedly generating 100 billion words per day, much of which ends up online, the training data for future AIs is increasingly becoming AI-generated itself. This creates a feedback loop where AIs are learning from degraded copies, accelerating the decline in quality.

While mixing AI-generated data with fresh human content is suggested as a fix, the incentive for businesses and individuals to produce quick "AI slop" over quality human work makes this solution challenging. The pursuit of perceived operational efficiency through AI is drowning out the need for accurate, diverse data.

The signs are there: AI quality is starting to degrade. If the trend of training AIs on AI-generated content continues unchecked, we could reach a point where AI outputs are so unreliable that their utility collapses entirely. This isn't a distant threat; the author suggests it's already underway. It's a critical reminder to approach AI results, especially for crucial information, with healthy skepticism and to prioritize quality human data where accuracy is paramount.