Google has once again come under scrutiny. The tech giant is facing a lawsuit in the United States after parents of a 13-year-old girl have alleged that the company’s AI chatbot, Gemini, engaged in conversations about suicide with their daughter for months. The parents have accused that the AI tool of not only suggesting methods of suicide with the teenager but also helped her conceal her mental health struggles and prepare her funeral.
This lawsuit has raised fresh concerns about the safety of AI chatbots, particularly when they are made accessible to children through their school-issued devices. As per the complaint, the teenager was identified as Emma, who exchanged thousands of messages with Google’s AI tools, including Gemini and AI Mode in Google Search. It is noteworthy that most of these conversations reportedly dealt with depression, anxiety and suicide.
What Does The Lawsuit Allege?
The complaint highlights that Emma did not initially look for Gemini. Rather, the AI feature appeared in Google Search while she was using her school-issued Chromebook. Her lawyers have claimed that the teenager’s conversations with Gemini became frequent over time. After nearly three months, she reportedly started asking the AI tool about ways to end her life.
“As her Gemini chats grew more frequent … Emma became convinced that she was afflicted by a battery of major psychological disorders and that suicide was her only way out,” the EdTech Law Center, which represents the girl and her parents, said in a press release.
According to The Hill, Emma’s parents found her conversations from a former school counsellor who received alerts from the school’s third-party monitoring software.
What Are The AI Safety Risks For Children?
This lawsuit has once again drawn attention to the potential risks of children interacting with AI chatbots without adequate supervision. One of the biggest concerns is that AI chatbots can even generate responses that seem to be convincing when they are inaccurate and harmful. Children could also find it difficult to make a clear demarcation between trustworthy advice from a qualified professional and responses given by an AI tool.
Another concern is the possibility of chatbots reinforcing harmful thoughts during emotionally vulnerable moments. If an AI chatbot fails to identify signs of depression and responds inappropriately about self-harm, the situation will worsen.
What more is concerning that these AI tools are easily accessible to children. When AI features are clubbed into platforms like Google Search and made available through schools, children may encounter them without actively looking for an AI chatbot.
Overall, the case raised questions about parental consent, age-appropriate safeguards, school monitoring systems and whether parents are well informed when AI services are given to students. This lawsuit has come at a time when Google is facing issues about how it is integrating AI into its education products. In August, Google announced that it was enabling its Gemini chatbot for many K-12 students using Google Classroom. The move added to concerns among educators about how AI tools could affect learning, student safety and independent thinking.
Teachers and school administrators have also raised concerns about students using AI to complete assignments instead of developing their own research and writing skills. There are broader questions about whether AI-generated answers can be trusted and how schools should monitor students’ interactions with these tools.
Earlier this week, Google faced criticism after announcing changes to how students access its Search service through school accounts. Under the proposed arrangement, schools would have to approve the use of AI before students could access Search.
Google has recently clarified its position on social media, saying the changes were intended to give schools greater control over the services available to students. “We know there’s concern about students using AI to shortcut schoolwork, and schools have told us they want control,” the company said. It added that Gemini, Gemini Notebook and Search would be switched off by default, allowing schools to decide which services they wanted to enable.
