AI Surveillance in Schools Sparks Arrests Over Misinterpreted Jokes and Private Messages

Schools across the United States are increasingly deploying AI-driven surveillance tools to monitor students’ online communications, often leading to unintended and disproportionate consequences. In one case, a 13-year-old girl from Tennessee was arrested after making a racially offensive joke in a private online chat with classmates. The comment, a response to being teased about her tanned skin, read: “on Thursday we kill all the Mexico’s.” The school’s monitoring system flagged the message, triggering a rapid response from law enforcement. The girl was interrogated, strip-searched, and placed in a jail cell for the night [1].

Her mother, Lesley Mathis, criticized the system, noting the lack of context in the AI’s interpretation. “It made me feel like, is this the America we live in?” she said. The girl’s arrest culminated in eight weeks of house arrest, psychological evaluation, and 20 days at an alternative school [1]. Mathis said her daughter later told her, “I thought you hated me,” a moment that “haunts you.”

Surveillance tools like Gaggle and Lightspeed Alert are used by thousands of school districts nationwide to detect signs of self-harm, bullying, or violence. Tennessee’s 2023 zero-tolerance law required immediate law enforcement notification for any threats of mass violence against schools, contributing to the escalation in such cases. Gaggle’s CEO, Jeff Patterson, stated the school in question did not use the tool as intended, which is to identify early warning signs and facilitate intervention before law enforcement is involved [1].

Similar incidents have occurred elsewhere. In Florida, a teenager was arrested for making a joke about school shootings on Snapchat, which was flagged by the platform’s automated detection software. Another student, Alexa Manganiotis, described how a surveillance tool at her school, Lightspeed Alert, detected a threatening comment that had already been deleted. Within minutes, the students were removed from class and taken away [1].

Critics argue that such systems disproportionately target children for speech that would likely go unpunished in adults. “If an adult makes a super racist joke that’s threatening on their computer, they can delete it, and they wouldn’t be arrested,” Manganiotis said [1]. Amy Bennett of Lightspeed Systems defended the technology, stating it allows understaffed schools to be “proactive rather than punitive” [1].

However, a review of data from the Lawrence, Kansas school district found that nearly two-thirds of the 1,200 Gaggle alerts over 10 months were deemed nonissues. These included false positives such as student homework, flagged photography class assignments, and a college essay edited by a student. Natasha Torkzaban, a 2024 graduate, was flagged for editing a friend’s college essay that included the phrase “mental health.” She and other students later filed a lawsuit against the school system, alleging unconstitutional surveillance [1].

Despite the high rate of false alarms, school officials argue the technology has also prevented potential crises. In Florida’s Polk County Schools, nearly 500 Gaggle alerts over four years led to 72 involuntary hospitalizations under the Baker Act. However, mental health experts caution that such interventions can be traumatic for children. “A really high number of children who experience involuntary examination remember it as a really traumatic and damaging experience,” said Sam Boyd of the Southern Poverty Law Center [1].

Jeff Patterson of Gaggle reiterated that the technology should be used for early intervention rather than punitive action. “I wish that was treated as a teachable moment, not a law enforcement moment,” he said [1]. While schools like Mathis’ daughter’s alternative school have taken steps to support students’ mental health, many families remain skeptical of the broader implications of AI surveillance in education.

Mathis said her daughter is doing better two years later but is still “terrified” of running into one of the school officers who arrested her. The experience has left her questioning the balance between safety and surveillance in schools. “It’s like we just want kids to be these little soldiers, and they’re not,” she said. “They’re just humans.” [1]

Source: [1] Schools are using AI to spy on students and some are getting arrested for misinterpreted jokes and private conversations (https://fortune.com/2025/08/07/schools-ai-surveillance-students-children-arrested-jokes/)