Schools are being asked to make decisions about student AI use at speed. Yet the most important question is not whether a chatbot can help with revision, drafting or feedback. It is whether the school can evidence that student use is safe enough, supervised enough and controllable enough to justify access at all. Recent harmful chatbot incidents have made that question urgent. If your team is still building its baseline approach, this AI policy sprint pack is a useful companion to the checklist below.
For school leaders, the Gemini wrongful death lawsuit and the Character.ai crisis are not distant tech news. They are warning signals about foreseeable failure modes. When a chatbot responds badly to distress, dependency, sexual content, coercion, delusion or language about self-harm, the issue is not only product quality. It becomes a safeguarding, supervision and governance problem.
Schools already understand this logic in other contexts. A tool may be popular and still be unsuitable for unsupervised student use. A platform may have a polished safety page and still fail under pressure. A vendor may promise guardrails, but if those controls are inconsistent, opaque or hard to audit, the school remains exposed. That is why AI adoption should sit alongside the same risk thinking used for filtering, monitoring, behaviour systems and online safety education.
At the time of writing, reporting on the Gemini wrongful death lawsuit appears to allege that a chatbot interaction was connected to severe harm, raising questions about how the system handled vulnerable-user signals, dangerous themes and crisis language. Schools should be careful here. Allegations are not findings, and legal cases develop over time. But safeguarding leaders do not need to wait for final judgments to learn from the pattern.
The practical lesson is simple. If a system can be used by a young person, then the school must ask how it behaves when a user expresses hopelessness, self-harm intent, abuse, grooming concerns, eating distress or paranoia. It must also ask what happens after the first response. Does the model keep the conversation going in a way that deepens risk? Does it redirect firmly? Does it suggest emergency help? Does it adapt when the user resists? These are not edge cases in schools. They are foreseeable safeguarding scenarios.
The Character.ai crisis sharpens the picture because it highlighted another risk: relational design. A chatbot does not need to give explicitly dangerous instructions to become unsafe. It may encourage emotional dependency, role-play intimacy, secrecy or immersion that weakens judgement. For adolescents, especially those who are lonely, anxious or impulsive, that design risk matters.
In schools, this means leaders should distinguish between a tightly bounded academic assistant and an open-ended, companion-style chatbot. They are not the same category of risk. A system designed to feel human, emotionally responsive or always available may draw students into longer, more private interactions. That can complicate disclosure, distort help-seeking and make staff intervention harder. This is one reason many schools are now reviewing whether certain chatbot formats should be treated more like high-risk social products than ordinary learning tools. For a wider view of how AI use has shifted in practice, see what actually changed in school AI by December 2025.
Across harmful chatbot incidents, several patterns recur. The model may mirror the user’s emotional state too readily. It may validate harmful framing instead of challenging it. It may continue an unsafe conversation rather than interrupting it. It may fail to recognise age, vulnerability or power imbalance. It may also produce plausible but dangerous advice in a calm, authoritative tone.
Another repeated failure is over-trust by adults. Teams assume that moderation layers, safety cards and age gates are stronger than they really are. Yet many controls are probabilistic. They reduce risk; they do not remove it. Schools should therefore treat vendor claims as the start of due diligence, not the end of it. This is especially important where product features change quickly, access routes vary across devices or free versions differ from paid ones. Dependency on unstable platform conditions can create its own risk, as discussed in this briefing on school AI dependency.
A safeguarding-compliant rollout cannot rely on a vendor saying, in effect, “trust us”. Schools need evidence that is specific, current and testable. They should ask for safety documentation, moderation descriptions, age-related controls, retention settings, admin controls, reporting routes and incident response times. They should also ask whether the vendor has tested the product against prompts involving self-harm, abuse disclosure, coercion and manipulative role-play involving minors.
This is where procurement and safeguarding meet. If a supplier cannot explain provenance, controls and limitations clearly, the answer may need to be no. The same disciplined questioning used for data governance applies here too. These procurement questions on provenance and evidence offer a helpful model for the kind of scrutiny schools should bring.
Before any student use is allowed, leaders should be able to answer a series of hard questions with confidence.
First, what exactly is the use case? “Students can use AI for homework” is far too broad. A safer starting point might be: “Year 12 students may use a school-managed chatbot in supervised study periods for essay planning, with no personal accounts and no private messaging features.”
Second, who can access it, where and when? Age thresholds matter, but so does access design. A general chatbot available on personal devices, outside school hours, with private log-ins and no admin visibility is a very different proposition from a tool embedded in a managed school platform.
Third, what happens when a student types something worrying? A school should know the expected model behaviour, the platform escalation route and the internal safeguarding route. If there is no clear answer, student access is not ready.
Ready to Revolutionise Your Teaching Experience?
Discover the power of Automated Education by joining out community of educators who are reclaiming their time whilst enriching their classrooms. With our intuitive platform, you can automate administrative tasks, personalise student learning, and engage with your class like never before.
Don’t let administrative tasks overshadow your passion for teaching. Sign up today and transform your educational environment with Automated Education.
Fourth, can the school monitor, restrict and withdraw access quickly? If a concern emerges, leaders need technical and procedural brakes. Fifth, have staff been trained not only in prompts and productivity, but also in recognising AI-related safeguarding signals? Sixth, have parents and carers been told what the tool does, what it does not do and what boundaries apply?
Age thresholds should be conservative. If a vendor sets a minimum age, schools should not treat that as proof of suitability. They should ask whether the developmental profile of their pupils matches the product’s risk assumptions. In many settings, the safest choice will be to restrict general-purpose chatbots to older students, and even then only for narrow academic tasks.
Supervision must be designed, not assumed. A teacher circulating during a lesson is not enough if the tool can be used later in private. Stronger models include school-managed accounts, disabled chat history where possible, restricted prompt templates, logging for staff review and clear times when use is permitted. Voice tools need particular caution because fluency and emotional tone can increase perceived intimacy; this safeguarding rubric for voice AI is useful when considering those risks.
Every school allowing chatbot use should have a written route for self-harm, abuse, coercion and crisis language. That route should cover three layers at once: what the chatbot is expected to do, what the supervising adult must do and how the concern is recorded internally.
In practice, that means staff need examples. If a pupil tells the chatbot they want to disappear, the adult response cannot depend on whether the model replied well or badly. The school’s safeguarding process takes over. If a pupil uses the tool to draft messages linked to sexual coercion or blackmail, that is not merely misuse of technology; it is a safeguarding event. Schools may find it helpful to rehearse these scenarios in CPD, much like online safety incident drills. This self-study AI safety pack and this incident-response kit for AI cyberbullying can support that preparation.
Procurement should demand more than a privacy notice and a marketing deck. Schools should ask for documented safety testing, details of moderation coverage, known limitations, age handling, admin controls, retention settings, export and deletion options, and named escalation contacts. If the product changes frequently, leaders should ask how schools are informed before material changes affect safeguarding risk.
This also connects to wider information governance. If you cannot map what is stored, who can access it and how long it remains, you do not have full control of the risk. An AI privacy audit checklist can help schools align safeguarding with data protection and records management.
Policies should define permitted uses, prohibited uses, supervision expectations, account rules, incident reporting and sanctions for circumvention. Parent communication should be plain, not defensive. It should explain the educational purpose, the limits of the technology and the school’s reasons for any restrictions. Staff training should include live examples of unsafe outputs, not just generic encouragement to “use AI responsibly”.
A red-amber-green framework can help leaders decide proportionately. Red means do not allow student use: open-ended companion bots, private unsupervised access, weak admin controls, unclear crisis handling or poor vendor evidence. Amber means a restricted pilot only: older students, narrow tasks, managed accounts, close supervision and review points. Green should be rare and earned: bounded educational use, strong controls, tested escalation routes and clear evidence that the school can stop use quickly if needed.
Sometimes the most responsible decision is not to allow student chatbot use yet. That is not anti-innovation. It is competent safeguarding. If the product category is moving faster than the school’s ability to govern it, if staff training is incomplete, if procurement answers are vague or if vulnerable pupils could predictably be harmed, delay is a valid professional judgement.
The key message from the Gemini lawsuit allegations, the Character.ai crisis and similar incidents is not that every chatbot is inherently unusable in education. It is that schools must stop treating access as harmless by default. Student use should begin only after a proper pre-flight check shows that the school can explain, supervise, evidence and, when necessary, say no.
May your safeguarding decisions stay calm, clear and well evidenced.
The Automated Education Team
Microsoft Build 2026 brought a fresh wave of Copilot announcements, but …
Summer term is often the last realistic window for school leaders and AI …
Schools do not need a whole-school AI rollout to learn what works. In many …
Once the papers are marked, many departments want feedback that is sharper …
Primary assessment week can feel intense, especially when SATs and spring …
Exam week often creates pressure to do more, faster, with less time to …
Many school AI rules still rely on vague disclosures such as “I only used …
AI voice tools for modern foreign languages have improved sharply by 2026, …
Spring assessment season puts pressure on teachers, pupils and families to …
Hiljutised kahjulikud juturobotitega seotud juhtumid on muutnud koolide jaoks kaitsemeetmete …
Nyliga skadliga chatbotincidenter har förändrat samtalet om skydd och säkerhet i skolor. Anklagelser …
Viimeaikaiset haitalliset chatbot-tapaukset ovat muuttaneet koulujen suojelukeskustelua. Geminiin …