We have all seen it happen. You ask an AI a question, and it gives you a confident, well-structured answer that sounds completely reasonable. But something feels off. You double-check, and it turns out the AI was wrong. Not just slightly wrong — confidently, elaborately wrong.
This phenomenon has a name, and new research is shedding light on why it happens and what it means for businesses that rely on AI.
A paper titled “Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution” reveals something troubling: AI systems can produce reasoning that looks perfectly logical on the surface, yet the logic doesn’t actually connect to the facts.
The “Right Answer, Wrong Reasoning” Problem
The researchers designed a clever test. They took an AI’s chain-of-thought reasoning — the step-by-step explanation it gives for how it reached a conclusion — and surgically replaced one of the premises with a nonsense symbol. If the AI’s reasoning was genuinely based on that premise, the conclusion should change. If not, there’s a disconnect.
The results were eye-opening. The researchers found that when they substituted a key premise, the AI’s conclusion often stayed the same — even though the AI claimed to be using that premise in its reasoning. In effect, the AI was producing the right answer for the wrong reasons, or worse, the wrong answer with a seemingly flawless justification.
This is not a rare edge case. The study found that 66% of correctly-solved problems contained at least one reasoning step that didn’t genuinely depend on the premise it claimed to be using. That’s two out of every three correct answers.
What This Means for Your Business
For Irish small business owners, this research has immediate practical implications. If you are using AI to draft contracts, prepare tax filings, write marketing copy, or analyse financial data, you cannot assume that a confident, well-reasoned answer is a correct one.
Imagine you ask an AI to review a lease agreement and flag any problematic clauses. It comes back with a detailed analysis, referencing specific sections of the lease and explaining why they are risky. It looks thorough and professional. But as the research shows, the AI might be generating an explanation that sounds plausible without actually being grounded in the document it analysed.
The same applies to AI-generated financial reports, customer communications, or compliance documents. The AI may produce something that looks perfect but contains hidden errors — errors that a human reader might not catch because the presentation is so convincing.
How to Protect Your Business
This doesn’t mean you should abandon AI tools. The benefits are real: speed, efficiency, and the ability to handle tasks that would take humans much longer. But it does mean you need to use AI with a healthy dose of skepticism. Here’s how:
1. Always verify AI outputs against source material. If an AI claims to have analysed a document, check the specific references. Does the document actually say what the AI claims it says?
2. Look for “too perfect” answers. Be suspicious of AI responses that are unusually polished or confident, especially on complex topics. The most dangerous AI errors are the ones that look right.
3. Use AI for drafts, not final decisions. Treat AI outputs as a starting point, not an endpoint. Your human judgment is still the most important quality control mechanism.
4. Train your team. Make sure everyone in your business who uses AI understands that it can produce confident-sounding but incorrect reasoning. A well-trained team is your best defence.
Why This Research Matters
The EU AI Act places responsibility for AI outputs on the organisations that deploy them. If your business uses AI to generate customer-facing content or make operational decisions, you are ultimately responsible for the accuracy of those outputs. Understanding that AI can be confidently wrong is not just a technical curiosity — it is a compliance necessity.
The researchers developed a test that can detect this kind of “right answer, wrong reasoning” problem. Their method achieved 88.5% accuracy at detecting when an AI’s reasoning was genuinely grounded in its premises — significantly better than previous approaches. As these audit tools become available commercially, they will become an important part of any business’s AI quality control toolkit.
For now, the lesson is simple: trust your AI, but verify. A confident answer is not always a correct one, and a well-reasoned explanation is not always grounded in fact. Irish businesses that understand this will avoid the pitfalls that catch the unwary.