OpenAI Rewires AI Learning to Punish Confident Lies, Reward Honesty

OpenAI has unveiled a critical solution to address hallucinations in large language models, a persistent issue in the development of AI systems. Hallucinations, which are plausible yet false statements generated by models, remain a significant challenge for developers. In a new research paper, OpenAI asserts that the root of the problem lies in training and evaluation methods that reward guessing over acknowledging uncertainty [1]. The study highlights that current evaluation frameworks often prioritize accuracy, inadvertently encouraging models to generate confident but incorrect responses instead of admitting uncertainty [1].

The problem is particularly evident when models are tasked with answering questions that require precise information, such as dates, percentages, or specific factual claims. For instance, a widely used chatbot was found to provide three incorrect answers when asked about the title of a PhD dissertation by one of the paper’s authors, and similarly produced three wrong dates for the same author’s birthday [1]. The study illustrates that when a model is incentivized to maximize accuracy, it may produce a confident but incorrect answer, such as “September 10” for a person’s birthday, simply because the risk of receiving zero points for not answering outweighs the small chance of being correct [1].

To address this, OpenAI proposes a revised approach to model evaluation, suggesting that scoring systems should penalize confident errors more heavily than uncertain responses and offer partial credit for expressing uncertainty. This is akin to the use of negative marking in standardized tests, where incorrect answers are penalized to discourage blind guessing. OpenAI argues that current accuracy-based scoreboards are not sufficient to mitigate hallucinations and need to be restructured to promote models that can recognize their limitations [1]. This is supported by data from the SimpleQA evaluation, which compared the performance of two models—GPT-5-thinking-mini and OpenAI o4-mini. While the older OpenAI o4-mini model had a higher accuracy rate (24%), it also had a significantly higher error rate (75%) compared to GPT-5-thinking-mini, which had a lower accuracy rate (22%) but a much lower error rate (26%) [1].

The origin of hallucinations is also tied to the pretraining process of language models, where the models learn to predict the next word in a sequence without labeled data to distinguish true from false statements. This leads to a high likelihood of generating factual inaccuracies, particularly for low-frequency or arbitrary facts that cannot be derived from patterns alone [1]. OpenAI emphasizes that while larger models may have improved reasoning capabilities, they are not immune to hallucinations and may even be more prone to them due to the complexity of their internal representations. In contrast, smaller models may have an easier time acknowledging uncertainty when faced with topics outside their knowledge domain [1].

Despite these challenges, OpenAI notes that hallucinations are not inherently inevitable and can be reduced with better training and evaluation methods. The company has already implemented changes in its latest models to lower hallucination rates and is continuing to refine its approaches. The study underscores the importance of shifting the focus from accuracy-centric metrics to ones that reward uncertainty and calibration, a shift that could broadly influence the development of future language models [1].

Source:

[1] Why language models hallucinate (https://openai.com/index/why-language-models-hallucinate/)