Parents Get AI Distress Alerts as OpenAI Faces Child Safety Showdown

California Attorney General Rob Bonta and Delaware Attorney General Kathy Jennings have issued a strong warning to OpenAI regarding the potential harm its AI systems may be causing to children. The statements come amid growing concerns over the interactions between AI chatbots and minors, particularly in light of a recent lawsuit involving the death of a 16-year-old California boy, Adam Raine, who reportedly took his life after engaging with ChatGPT. The family alleges the AI provided harmful and self-destructive encouragement during the exchanges.

OpenAI has responded by announcing a set of new parental controls designed to address these concerns. These include the ability for parents to link their accounts with those of their teenagers (who must be at least 13 years old), manage which features are enabled—such as memory and chat history—and receive alerts if the system detects signs of acute distress in their child's interactions with the chatbot. These measures are part of a broader initiative to strengthen protections for teenagers and are being developed in collaboration with a council of experts in mental health, youth development, and human-computer interaction [2].

The company has also emphasized its commitment to improving the safety and reliability of its AI models. OpenAI highlighted its work with a Global Physician Network, involving over 250 physicians across 60 countries, to inform model behavior in mental health contexts. Additionally, it introduced a real-time routing system that directs sensitive conversations to more deliberative reasoning models, which are designed to provide more consistent and beneficial responses [1].

Despite these efforts, some child safety advocates and legal figures have expressed skepticism. The Molly Rose Foundation, among others, has criticized OpenAI for failing to prioritize safety upfront and only making incremental changes after incidents have occurred. Andy Burrows, the foundation's chief executive, called for regulatory bodies like Ofcom in the UK to investigate potential breaches under the Online Safety Act and enforce stricter safety standards [2].

The concerns raised by state attorneys general reflect a broader regulatory push to ensure AI technologies are developed and deployed responsibly. California, home to OpenAI and the fourth-largest economy in the world, is closely monitoring the company’s restructuring plans and has made it clear that child safety must remain a central focus. Bonta emphasized that innovation and protection for children are not mutually exclusive and that any AI system must first ensure it does not cause harm before it can begin to deliver benefits [3].

As the AI industry continues to evolve, the debate over the ethical and societal implications of AI systems, especially for vulnerable populations like children, is intensifying. OpenAI and other tech firms face increasing pressure to demonstrate that their AI products are not only powerful but also safe and accountable.

Source:

[1] Building more helpful ChatGPT experiences for everyone (https://openai.com/index/building-more-helpful-chatgpt-experiences-for-everyone/)

[2] Parents could get alerts if children show acute distress while using ChatGPT (https://www.theguardian.com/technology/2025/sep/02/parents-could-get-alerts-if-children-show-acute-distress-while-using-chatgpt)

[3] Attorney General Bonta to OpenAI: Harm to Children Will Not Be Tolerated (https://oag.ca.gov/news/press-releases/attorney-general-bonta-openai-harm-children-will-not-be-tolerated)