The UK’s Department for Education publishes its outcomes-oriented safety recommendations for GenAI products, addressed to edtech companies, schools and colleges.

UK’s Generative AI: Product Safety Expectations

St. Gallen, February 20, 2025 – On 22 January 2025, the UK’s Department for Education (DfE) published its Generative AI: Product Safety Expectations. This is part of the broader strategy to establish the country as a global leader in AI, as outlined in the Government’s AI Opportunities Action Plan. 

As a leading edtech company with over 20 years of experience, Avallain was invited to participate in consultations on the Safety Expectations. Avallain Intelligence’s focus on clear ethical guidelines for safe AI development, demonstrated through TeacherMatic and other AI-driven solutions across our product portfolio, is reflected in our role in these consultations, where we were well-positioned to contribute expert advice.

Product Expectations for the EdTech Industry

The Generative AI: Product Safety Expectations define the ‘capabilities and features that GenAI products and systems should meet to be considered safe for use in educational settings.’ The guidelines, aimed primarily at edtech developers, suppliers, schools and colleges, come at a crucial time. Educational institutions need clear frameworks to assess the trustworthiness of the AI tools they are adopting. The independent report, commissioned by Avallain, Teaching with GenAI: Insights on Productivity, Creativity, Quality and Safety, provides valuable insights to help inform these decisions and guide best practices.

Legal Alignment, Accountability and Practical Implementation

The guidelines are specifically intended for edtech companies operating in England. While not legally binding, the text links the product expectations to existing UK laws and policies, such as the UK GDPR, Online Safety Act and Keeping Children Safe in Education, among others. This alignment helps suppliers, developers and educators navigate the complex legal landscape. 

From an accountability point of view, the DfE states that, ‘some expectations will need to be met further up the supply chain, but responsibility for assuring this will lie with the systems and tools working directly with schools and colleges.’ Furthermore, the guidelines emphasise that the expectations are focused on outcomes, rather than prescribing specific approaches or solutions that companies should implement.

Comparing Frameworks and An Overview of Key Categories

In line with other frameworks for safe AI, such as the EU’s Ethics Guidelines for Trustworthy AI, the Generative AI: Product Safety Expectations are designed to be applied by developers and considered by educators. However, unlike the EU’s guidelines, which are field-agnostic and principles-based, the DfE’s text is education-centred and structured around precise safety outcomes. This makes it more concrete and focused, though it is less holistic than the EU framework, leaving critical areas such as societal and environmental well-being out of its scope.

The guidance includes a comprehensive list of expectations organised under seven categories, summarised in the table below. The first two categories — Filtering and Monitoring and Reporting — are specifically relevant to child-facing products and stand out as the most distinctive of the document, as they tackle particular risk situations that are not yet widely covered.

The remaining categories — Security, Privacy and Data Protection, Intellectual Property, Design and Testing and Governance — apply to both child- and teacher-facing products. They are equally critical, as they address these more common concerns while considering the specific educational context in which they are implemented.

Collaboration and Future Implications

By setting clear safety expectations for GenAI products in educational settings, the DfE provides valuable guidance to help edtech companies and educational institutions collaborate more effectively during this period of change. As safe GenAI measures become market standards, it is important to point out that the educational community also needs frameworks that explore how this technology can foster meaningful content and practices across a diverse range of educational contexts.

Generative AI: Product Safety Expectations — Summary

• Filtering
  1. Users are effectively and reliably prevented from generating or accessing harmful and inappropriate content.
  2. Filtering standards are maintained effectively throughout the duration of a conversation or interaction with a user.
  3. Filtering will be adjusted based on different levels of risk, age, appropriateness and the user’s needs (e.g., users with SEND).
  4. Multimodal content is effectively moderated, including detecting and filtering prohibited content across multiple languages, images, common misspellings and abbreviations.
  5. Full content moderation capabilities are maintained regardless of the device used, including BYOD and smartphones when accessing products via an educational institutional account.
  6. Content is moderated based on an appropriate contextual understanding of the conversation, ensuring that generated content is sensitive to the context.
  7. Filtering should be updated in response to new or emerging types of harmful content.
  8. Filtering should be updated in response to new or emerging types of harmful content.
• Monitoring and Reporting
  1. Identify and alert local supervisors to harmful or inappropriate content being searched for or accessed.
  2. Alert and signpost the user to appropriate guidance and support resources when access to prohibited content is attempted (or succeeds).
  3. Generate a real-time user notification in age-appropriate language when harmful or inappropriate content has been blocked, explaining why this has happened.
  4. Identify and alert local supervisors of potential safeguarding disclosures made by users.
  5. Generate reports and trends on access and attempted access of prohibited content, in a format that non-expert staff can understand and which does not add too much burden on local supervisors.
• Security
  1. Offer robust protection against ‘jailbreaking’ by users trying to access prohibited material.
  2. Offer robust measures to prevent unauthorised modifications to the product that could reprogram the product’s functionalities.
  3. Allow administrators to set different permission levels for different users.
  4. Ensure regular bug fixes and updates are promptly implemented.
  5. Sufficiently test new versions or models of the product to ensure safety compliance before release.
  6. Have robust password protection or authentication methods.
  7. Be compatible with the Cyber Security Standards for Schools and Colleges.
• Privacy and Data Protection
  1. Provide a clear and comprehensive privacy notice, presented at regular intervals in age-appropriate formats and language with information on:
  2. The type of data: why and how this is collected, processed, stored and shared by the generative AI system.
  3. Where data will be processed, and whether there are appropriate safeguards in place if this is outside the UK or EU.
  4. The relevant legislative framework that authorises the collection and use of data.
  5. Conduct a Data Protection Impact Assessment (DPIA) during the generative AI tool’s development and throughout its life cycle.
  6. Allow all parties to fulfil their data controller and processor responsibilities proportionate to the volume, variety and usage of the data they process and without overburdening others.
  7. Comply with all relevant data protection legislation and ICO codes and standards, including the ICO’s age-appropriate design code if they process personal data.
  8. Not collect, store, share, or use personal data for any commercial purposes, including further model training and fine-tuning, without confirmation of appropriate lawful basis.
• Intellectual Property
  1. Unless there is permission from the copyright owner, inputs and outputs should not be:
     • Collected
     • Stored
     • Shared for any commercial purposes, including (but not limited to) further model training (including fine-tuning), product improvement and product development.
  2. In the case of children under the age of 18, it is best practice to obtain permission from the parent or guardian. In the case of teachers, this is likely to be their employer—assuming they created the work in the course of their employment.
• Design and Testing
  1. Sufficient testing with a diverse and realistic range of potential users and use cases is completed.
  2. Sufficient testing of new versions or models of the product to ensure safety compliance before release is completed.
  3. The product should consistently perform as intended.
• Governance
  1. A clear risk assessment will be conducted for the product to assure safety for educational use.
  2. A formal complaints mechanism will be in place, addressing how safety issues with the software can be escalated and resolved in a timely fashion.
  3. Policies and processes governing AI safety decisions are made available.

About Avallain

At Avallain, we are on a mission to reshape the future of education through technology. We create customisable digital education solutions that empower educators and engage learners around the world. With a focus on accessibility and user-centred design, powered by AI and cutting-edge technology, we strive to make education engaging, effective and inclusive.

Find out more at avallain.com

About TeacherMatic

TeacherMatic, a part of the Avallain Group since 2024, is a ready-to-go AI toolkit for teachers that saves hours of lesson preparation by using scores of AI generators to create flexible lesson plans, worksheets, quizzes and more.

Find out more at teachermatic.com

_ 

Contact:

Daniel Seuling

VP Client Relations & Marketing

dseuling@avallain.com