Two governance problems, not one
Educational institutions face AI governance on two simultaneous fronts, and conflating them produces poor policy on both. The first is the question of how students may use AI in their academic work, a pedagogical and academic integrity question. The second is how the institution itself deploys AI in its operations, an organisational governance question with direct regulatory and duty of care implications.
Most institutional AI policy attention has focused on the first question. The second deserves equal attention, and carries more significant legal and regulatory exposure.
Institutional AI deployments: the high-risk problem
Universities and schools increasingly use AI systems in consequential decisions about students: admissions scoring, scholarship allocation, academic performance prediction, learning disability identification, and student wellbeing risk assessment. These applications are high-risk AI under the EU AI Act. Institutions with students from EU member states: which includes most English-speaking universities with international student populations, are directly subject to the Act's requirements for high-risk AI deployers.
The obligations are substantial. High-risk AI deployers must implement human oversight measures, conduct fundamental rights impact assessments, monitor AI system operation, and inform affected individuals that AI is being used in decisions about them. For admissions AI, this means students have rights to know AI was used in assessing their application, and in some circumstances to request human review.
Most institutions deploying AI in these contexts have not yet implemented the governance structures these obligations require.
AI academic integrity tools: a governance failure in progress
AI detection tools, software that claims to identify AI-generated student submissions, have become widely deployed in educational settings. The governance problem is that these tools have documented and significant false positive rates: they incorrectly identify human-written work as AI-generated at rates that are unacceptable for consequential academic decisions.
Multiple documented cases exist of students facing academic misconduct proceedings based on AI detection tool outputs, where the work was human-authored. In some cases, non-native English speakers have been disproportionately flagged, creating discrimination risk on top of accuracy concerns.
Governance requires treating AI detection tool outputs as investigative information, a starting point for inquiry, not as evidence sufficient to support an academic misconduct finding. No academic misconduct determination should be made on the basis of AI detection output alone, without corroborating human assessment of the work.
Student data and privacy
Student data is among the most sensitive categories of personal data, it includes minors in K-12 settings, health and disability information in student support contexts, and data about individuals at a formative stage of their lives. The privacy obligations that apply to this data significantly constrain how AI tools processing it can be used and what vendors can do with it.
In Australia, the Privacy Act and applicable state education legislation create obligations around student data collection and use. In the US, FERPA and COPPA apply. In the EU and for EU students, GDPR applies with particular force to processing of children's data.
Vendor contracts for AI tools used in educational settings must be reviewed against applicable student privacy law before deployment. Vendors who seek to use student data for model training, who retain data beyond the term of the contract, or who share data with third parties without appropriate controls present unacceptable privacy risk in educational contexts.
AI in student-facing support roles
AI chatbots and virtual assistants are increasingly deployed in student-facing roles: academic advising, mental health triage, career guidance, and general student support. These deployments carry direct duty of care implications that institutional governance must address.
A student mental health support chatbot that fails to appropriately escalate a crisis situation, or that provides incorrect advice about academic withdrawal, is not simply a technology failure, it is an institutional failure with direct consequences for student welfare. Governance must establish clear boundaries for AI in student support roles, mandatory human escalation pathways, and ongoing monitoring of AI system performance in these sensitive contexts.
