Why Generative AI ‘hallucinates’ - and how CTOs can prevent it

It happens all the time: you’ll ask Gemini or ChatGPT a question and the response is confident, fluent, detailed - and completely wrong. This is "AI hallucination." 

It occurs when an AI model generates outputs that appear plausible but are completely fabricated, with no basis in reality. For CTOs, it’s an intriguing and disruptive challenge, with the potential to facilitate major errors in employee or even board-level decision-making. 

So how can CTOs prevent generative AI hallucinations and ensure outputs are more accurate? Let’s find out. 

How do AI hallucinations happen?

AI hallucinations occur where large language models produce incorrect or misleading results. Generative AI models learn by identifying patterns in vast datasets they are trained on. However, the accuracy of their output hinges on the quality and completeness of this data. If the training data is flawed — it could be incomplete, biased, or inaccurate — the model may learn incorrect patterns. This leads to inaccurate predictions, or in other words, hallucinations.

In a well-known example, one generative AI chatbot was asked about new discoveries from the James Webb Space Telescope to share with a child, the chatbot falsely claimed that the telescope captured the first images of an exoplanet. This was demonstrably untrue. NASA confirmed that the first exoplanet images were taken in 2004, long before the James Webb Telescope's 2021 launch.

Understanding the risks 

Industry experts are increasingly concerned about the ethical implications of AI models which generate inaccurate or misleading information, eroding user trust in the technology and potentially perpetuating the biases in the data on which they were trained. 

For business leaders, relying solely on data produced by an LLM can be disastrous. Potential consequences include heavily biased or uninformed decision-making, financial losses and reputational damage. 

It’s essential to remember that generative AI, while highly convincing, cannot understand the world as humans do. And for CTOs, this  raises critical questions about accountability. If human oversight is minimized or if users place excessive trust in AI, who is responsible when errors occur? 

How to mitigate AI hallucinations

CTOs overseeing AI models must be aware of the factors that contribute to these hallucinations. While complete eradication may be challenging, there are proactive steps to minimize their occurrence.

1. Improving the quality of training data: With most hallucinations occurring as a result of the model struggling in unfamiliar situations based on biased, incomplete or unrepresentative data, CTOs should invest in robust data collection and curation processes to ensure comprehensive, relevant, and representative data.
    
2. Adding guardrails: During the training process, limiting the range of possible outputs from the LLM can be beneficial. CTOs, in collaboration with their development teams, can guide the AI to focus on the most logical and promising responses, iterating continuously to reduce the chances of inconsistent or inaccurate outputs.
    
3. Vigorous testing & validation: QA is just as important in generative AI training as in shipping any other tech product. The goals are not only to improve performance but to allow for adjustments and retraining as datasets grow and evolve. Implementing a culture of rigorous testing and ongoing evaluation should be key to the CTO’s AI strategy.
    
4. Structuring outputs for clarity: Developers can create templates to guide the AI in presenting information in a specific format. A template for text generation, for example, could specify elements like a title, introduction, body, and conclusion, promoting structure and coherence.
    
5. Establishing ethical AI guidelines: CTOs must establish and enforce ethical guidelines for AI use within their organizations. This includes creating an AI governance framework that prioritizes responsible and fair AI practices, ensuring alignment with the company's values.
    
6. Maintaining Human Oversight: Generative AI should always be monitored by human oversight - particularly in business-critical decision-making. CTOs should define clear roles for human reviewers to interpret and validate AI-generated outputs.
    

Looking to the future

It’s true that, as generative AI continues to improve, hallucinations are likely to become less frequent. But they almost certainly can’t be prevented entirely. Just as humans will always be prone to error, so too are AI models. And as new models emerge, so too might new types of hallucination, requiring new approaches from business to mitigate the risks. 

CTOs must therefore focus on continuous improvement, recognizing that building trust in AI is an ongoing process. The goal is to create AI solutions that are not just nearly perfect but are truly reliable and integral to solving real-world challenges.

SBM can help. Contact us to learn how we can help your organization minimize risk in generative AI implementation and improve business outcomes.