Truthfulness is a real problem in AI. Not "sometimes wrong." Real. Structural. The kind of wrong that has a confident voice and a tidy paragraph structure and sources it invented.
The AI revolution sits on top of systems fundamentally resistant to change. Tax filings, compliance, app store approvals, legal documentation — these still eat your weeks. And AI can't do your visa paperwork. AI can't wait in the government queue for you. It can help you draft the cover letter, but the system it's submitting to still runs on fax-era logic.
On hallucinations: techniques like RAG and Chain-of-Thought are promoted as solutions. The problem persists. The fundamental issue is garbage input producing garbage output regardless of how clever the wrapper is.
They keep saying these things have intelligence that surpasses a PhD. I keep having to remind it of something I told it 15 minutes ago.
Read the full piece on Medium →