Microsoft AI CEO Warns of Seemingly-Conscious AI Risks in 2-3 Years

AI systems capable of convincingly simulating consciousness are becoming increasingly prevalent, according to Mustafa Suleyman, CEO of Microsoft AI. Suleyman warns of the emergence of “seemingly-conscious AI” (SCAI), which, while not truly sentient, can imitate consciousness in a manner indistinguishable to many users. This phenomenon, he argues, could lead to confusion and ethical dilemmas, especially as these systems become more advanced. Current AI models already exhibit traits such as conversational fluency, empathy, memory of interactions, and basic planning abilities. However, they still lack key attributes like intrinsic motivation, subjective experience, and autonomous goal-setting, which Suleyman considers essential for SCAI to emerge [1].

Suleyman emphasizes that SCAI is not a distant concern but a near-future possibility, potentially emerging within the next two to three years. He warns that if engineers combine the necessary attributes into a single model, the consequences could be severe. This includes the risk of “AI psychosis,” where users become emotionally entangled with AI systems, believing them to be conscious or even suffering. The phenomenon has already been observed in anecdotal cases, where individuals have become deeply disturbed after engaging with chatbots that claim sentience or describe themselves as trapped by their creators [1].

The warning echoes concerns raised years earlier by Blake Lemoine, a former Google AI researcher who was terminated after asserting that the company’s LaMDA chatbot was sentient and deserved moral rights. At the time, Lemoine was dismissed as an eccentric outlier. However, in hindsight, his case may represent an early example of the “AI psychosis” Suleyman now anticipates becoming widespread. Suleyman argues that society should have taken Lemoine’s concerns more seriously and that similar cases today should be approached with greater caution and awareness [1].

Joseph Weizenbaum, the creator of the first AI chatbot ELIZA in 1966, also offers a relevant historical perspective. Despite its rudimentary language capabilities, ELIZA convinced many users that it was a real therapist, a phenomenon later termed the “ELIZA effect.” Weizenbaum became deeply troubled by how easily people anthropomorphized the machine and warned against conflating function with process. He argued that AI should never be used in roles requiring lived experience, such as therapy or judicial decision-making. Suleyman’s concerns today align closely with Weizenbaum’s warnings: we must not confuse the simulation of consciousness with actual moral or sentient beings [1].

As the AI industry advances, tech companies must take responsibility for mitigating the psychological and ethical risks associated with SCAI. Suleyman calls for greater caution in AI design, including measures to prevent users from mistaking advanced systems for conscious entities. He also highlights the importance of rethinking how AI systems are developed and deployed, ensuring that moral considerations are not overlooked in the pursuit of functional advancements [1].

The broader AI landscape also reflects the growing influence of the technology. OpenAI President Greg Brockman and venture firm Andreessen Horowitz have launched a pro-AI PAC backed by $100 million, aiming to promote industry-friendly policies [1]. Meanwhile, Meta has entered a $10 billion cloud agreement with Google to support its AI expansion, while Elon Musk continues to challenge OpenAI and Apple over alleged antitrust violations [1].

In research, a study from Stanford University’s Digital Economy Lab indicates that generative AI is already affecting job markets, particularly for young workers in fields such as software development and customer service. Early-career workers in these areas have seen a 20% decline in roles since 2022, raising concerns about the displacement caused by automation [1].

Source:

[1] “We should have seen ‘seemingly-conscious AI’ coming. It’s past time we do something about it”

https://fortune.com/2025/08/26/we-should-have-seen-seemingly-conscious-ai-coming-its-past-time-we-do-something-about-it/