AI’s Causal Illusion: A Hidden Threat to Business Decisions

Artificial intelligence appears at the centre of many corporate strategy discussions. LLM tools like ChatGPT, Claude, Gemini, and Grok are being used by businesses for tasks from drafting documents to generating insights. Unfortunately, their ability to produce polished text has created the illusion that these tools genuinely understand what they’re writing about.

A recent paper, accepted for NeurIPS (Annual Conference on Neural Information Processing Systems), titled “Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?”, demonstrates that this is wrong. The research shows that despite their apparent fluency, these models lack a fundamental human capability: a genuine understanding of cause and effect.

For businesses who don’t recognise it, this gap is a serious strategic risk.

Sequences not causation

LLMs predict the next word in a sequence based on patterns they have learned from their training data. They get very good at sounding informed, but sounding plausible is not the same as understanding why things happen.

The research team used a new benchmark that was built from articles published after the models were trained. This meant the models were tested on truly new scenarios, not just regurgitating memorised information.

The results were poor. Claude 3’s accuracy dropped below 70%, LLaMA 2 was just over 50%, and GPT-3.5 struggled. Their supposed reasoning ability vanished when they couldn’t rely on familiar patterns.

The researchers identified two types of reasoning:

• Level-1 reasoning: this is just pattern matching, recalling known cause-and-effect relationships from the training data.
• Level-2 reasoning: is inferring new causal links from context, which is what humans do.

Today’s LLMs operate at Level-1. For instance, they ‘know’ that interest rate rises often come before a housing market slowdown because they’ve seen it many times in their training data, but they can’t explain why this happens or predict what would happen in new market conditions.

Why this matters

For a business, the risk isn’t just that the AI might be wrong. The real issue is that it delivers flawed analyses with complete confidence, potentially misleading decision-making.

This has a few key consequences:

• Strategic Planning: Flawed recommendations may be made, especially in uncertain or new situations.
• Risk Management: AI systems can’t reliably anticipate unintended consequences because they miss risks that don’t fit existing patterns.
• Accountability: Especially in regulated industries, decisions must be defensible. So ,if your AI can’t explain its reasoning, that could lead to legal problems.
• Innovation: Effective innovation requires that we understand why and how things work – LLMs’ do not give us this.

Realistic AI use

All of this doesn’t mean AI is useless; but it’s important to understand the limitations. LLMs are great tools for writing, summarizing, and handling information, but not for making logical arguments and choices.

For business leaders, there is a clear way forward:

• Human Oversight: Don’t let AI make critical decisions – Human experts are needed to validate.
• Know the Boundaries: Understand exactly what your AI tools can and cannot do.
• Horses for courses: Use AI for tasks it’s good at (e.g. drafting and summarising) and keep causal analysis and strategic reasoning for human experts.
• Test with Novelty: When evaluating AI, test it on genuinely new situations. Remember that performance on familiar benchmarks is misleading.

For now, view LLMs as very skilled imitators that can pretend to understand but don’t actually understand. They are useful for handling language, but they can’t replace human judgment.

Effective leadership means using AI with a clear consideration of what it can and can’t do.

Source:
Chi, H., Li, H., Yang, W., et al. (2024). Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?. Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS).