Podcast 146: AI systems do not hallucinate. Humans hallucinate, with Tony Fish

The rapid rise of AI has created a governance challenge that many organizations are only beginning to understand. In this conversation between Igor Tomych and fintech investor and advisor Tony Fish, the discussion explores why AI governance is becoming a structural issue for financial services. Rather than focusing only on regulation or model explainability, the episode examines deeper tensions between how banking systems operate and how modern technology platforms evolve.

AI development is moving faster than governance frameworks

Fish compares the current stage of AI to early technological revolutions such as electricity or steam power. At the time, societies adopted these technologies long before fully understanding their implications.

AI is following a similar trajectory. Companies are deploying models across decision-making processes while regulators and institutions are still debating how these systems should be governed. The challenge is not simply regulatory lag. It is that most governance frameworks assume stable, deterministic systems, while AI systems often behave probabilistically and adapt over time.

This creates a widening gap between technological capability and oversight capacity.

Banks operate like railways, tech companies like roads

One of the central ideas in the conversation is Fish’s comparison between banking infrastructure and technology systems.

Banks function like railways. Routes are predetermined, processes are tightly controlled, and outcomes must be predictable. This deterministic structure ensures reliability and auditability.

Technology companies operate more like roads. Decisions are made in real time, routes constantly change, and accountability is distributed among many actors.

When AI systems developed in technology environments enter banking systems designed for deterministic processes, governance challenges quickly emerge.

Money and trust create different risks than data and attention

Fish also highlights a fundamental economic difference between financial institutions and technology platforms.

Banks operate on money and trust, while technology companies operate on data and attention.

If a tech platform produces the wrong recommendation or advertisement, the consequences are limited. In financial services, incorrect decisions can have irreversible effects, from denied credit to misallocated funds. This makes reliability and governance significantly more important in financial contexts.

AI deployed in banking therefore carries different expectations than AI used in consumer technology platforms.

The real challenge is system complexity

Public debate around AI often focuses on explainability. Fish argues that this only addresses part of the problem.

Explaining a model’s decision does not necessarily mean the overall system is understandable. Modern financial infrastructure already includes legacy systems layered with multiple regulatory frameworks. When dozens of interacting AI models are added to this environment, emergent behaviors can appear that no single team fully understands.

The governance challenge therefore extends beyond individual models to the behavior of entire systems.

Regulatory models struggle to keep pace

Financial regulation traditionally relies on static audits and predefined compliance rules. These mechanisms work well when systems behave predictably.

AI systems complicate this approach. Their outputs evolve based on data and interactions with other models. Fish describes this as an “exponential asymmetry,” where AI capabilities develop faster than regulatory auditing processes.

As financial institutions deploy more AI-driven infrastructure, regulators may need entirely new governance approaches capable of monitoring adaptive systems rather than static ones.

Innovation requires both stability and rule-breakers

The episode concludes with a metaphor from nature.

In a bee colony, most bees follow the “waggle dance” to known food sources. However, a smaller group ignores the signal and searches for new resources. Many fail, but their exploration ensures the survival of the hive if existing food sources disappear.

Fish suggests organizations should function similarly. Most teams must follow established processes to maintain stability. But a smaller group must actively challenge assumptions and explore new approaches.