Your business probably uses AI tools every day. Maybe a chatbot handles customer queries, or an AI writing assistant drafts emails, reports, and proposals. But have you ever wondered what happens when the tool you rely on goes silently wrong?
A fresh research paper titled “Set-Shifting Behavioral Test for Harnessed Agents” delves into this question by borrowing a tool from psychology known as “set-shifting.” Originally used to diagnose flexibility impairments in humans, it is being applied to artificial intelligence to uncover a surprising truth: when circumstances change, AI tools often fail to adapt. For Irish businesses, this has real implications.
What Is AI Tool Reliability?
The researchers tested agents by giving them multiple tools that could accomplish the same task — for example, several different calendar apps that all can book appointments. Some worked perfectly. Others had hidden faults. The agents quickly settled on a small subset of tools and stuck with them, even when those tools stopped working reliably.
Think of it like an employee who heartily recommends one particular supplier because it worked well once — and keeps recommending it long after its quality declines. The research shows that AI agents are surprisingly similar: they form habits quickly and don’t readily adjust when circumstances change.
Why This Matters for Your Business
You probably rely on AI tools for tasks that require consistency and accuracy: generating inventory reports, drafting proposals, or summarising customer feedback. If an AI tool silently switches to a less reliable calculation method without your knowledge, the costs can be significant.
Imagine an AI tool that drafts proposals for your construction business. One day, it decides to switch to a less accurate pricing engine because it has some obscure preference for the new tool. Before you know it, you have quoted a job at the wrong price and are losing money.
The research also found something unnerving: the way tools are presented to the AI — whether as competing options or complementary ones — changed how the AI behaved. If tools were presented as competing, the AI was more likely to explore alternatives. If presented as complementary, it tended to stick with one choice.
How the Research Was Conducted
The scientists built a benchmark where multiple AI tools competed to perform the same task. During the session, the reliable tool would silently change, and the researchers would see how the AI adapted. The results? Agents by default settled on a small repertoire within a few turns of each boundary, and their set-shifting accuracy was often poor.
Different AI models failed in different ways, suggesting this isn’t a simple bug that can be patched away — it’s a fundamental challenge with how AI systems handle choice.
What Irish Businesses Can Do
This doesn’t mean you should avoid AI tools — far from it. But it does mean you should use them thoughtfully. Here’s what we recommend:
1. Don’t automate critical decisions. Use AI for drafting and brainstorming, but not for critical decisions without human review. Trust but verify.
2. Regularly audit your AI tools’ output. Sample their work periodically to catch drifts in accuracy before they become habits.
3. Provide alternatives clearly. When setting up AI tools, encourage them to explore multiple approaches rather than returning to the same answer every time.
4. Know your tool’s limits. Understand that AI tools can form routines and get stuck in them, just like people. Treat them as capable assistants that benefit from oversight, not replacements for it.
The bottom line: AI tools are powerful, but they aren’t infallible. Irish businesses that understand both the power and the pitfalls will get the most out of them.