The EU’s AI Act And Product Safety

• “Providers” placing on the market (or putting into service) in the EU AI systems, regardless of where they are based, and “users” of such systems within the EU.
• “Product manufacturers” applying high-risk AI systems to products shall take the responsibility of the compliance of the AI system and have the same obligations imposed by the present Regulation on the “Provider”.
• “Authorised representatives” established in the EU which have a written mandate from a provider of an AI system to carry out obligations outlined in the Act on its behalf.
• “Importers” that place on the market AI systems in the name or trademark of an entity outside the EU.
• “Distributors” that make available AI systems without affecting their properties.

• Definition of “Providers”: Providers are defined as “a natural or legal person, public authority, agency or other body that develops an AI system or that has an AI system developed with a view to placing it on the market or putting it into service under its own name or trademark, whether for payment or free of charge”.
• Definition of “Users”: Users are defined as “any natural or legal person, public authority, agency or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity”.
• Definition of “Product manufacturer”: The manufacturer of a product, subject to the acts listed at Annex II, sold under its name and applying a high-risk AI system.

1. Prohibited uses
2. High risk
3. Low or limited risk

1. Prohibited use
   Some forms of AI are explicitly prohibited under the AI Act as they are deemed to be an unacceptable level of risk and/or unacceptable use purposes. See further detail below.
2. High-Risk
   Some forms of AI are classed as high risk, where they:
   a.) are (1) intended to be used as a product covered by Union harmonisation legislation listed in Annex II (or a safety component of one), which covers products such as machinery, toys and medical devices, and (2) the product is required to undergo a third party conformity assessment with a view to placing on the market/putting into service; and/or
   b.) are listed specifically at Annex III (i.e. biometric ID, critical infrastructure safety, education and vocational training, employment, public benefits, law enforcement, border control and administration of justice).
3. Low/Limited Risk
   Other forms of AI not falling within the prohibited use and high-risk categories will be deemed low or limited risk. These are subject to much less prescriptive obligations. Generally speaking, this category includes spam filters, chat bots, and other non-intrusive forms of product.

• Those that deploy subliminal techniques to “materially distort a person’s behaviour in a manner that causes or is likely to cause … physical or psychological harm”.
• Those that exploit vulnerabilities of a specific group due to age or disability to cause physical or psychological harm.
• Those deployed by or for public authorities to classify or evaluate people based on behaviour or personality which could lead to scores which have detrimental effect on treatment of people/groups which are unrelated and/or disproportionate.
• Those involving real-time biometric ID in public spaces for law enforcement (unless strictly necessary for finding victims of crime or suspects of specified offences, or prevention of a specific substantial threat to life).

• Databases based on bulk scraping of facial images.
• Systems which categorise individuals based on sensitive personal traits such as race or political views.
• Predictive policing software to predict likelihood of crime based on personal traits.
• Emotion recognition in workplace/education environments, except where used for safety reasons.

• Required to undergo conformity assessment including drawing up of technical documentation and declaration of conformity before placing on the market.
• In certain limited use cases, third-party conformity assessment by a notified body is required, including in cases regarding biometric identification and categorisation system providers.
• Risk management system to be implemented and maintained.
• Testing throughout development and prior to placing on the market against defined metrics.
• Using only data which meets quality criteria to train models (where applicable).
• Design products with capabilities for automatic recording of event logs which ensure traceability of risk-related events as well as usage and input data.
• Inclusion of instructions for use which identify the Provider and any risks of use.
• Design for human oversight during their use.
• Design to achieve an appropriate level of accuracy, robustness and cybersecurity throughout its lifecycle.
• Products which continue to learn after their being placed on the market should be developed to address potential feedback loops and bias.
• Registration to an EU database, which is to be developed by the European Commission.
• Ensuring systems are designed to inform any natural person using it that they are using an AI system, and in the event they generate or manipulate image/audio content to create “deepfakes”, disclose the artificial generation/manipulation of the content.
• Undertake post-market monitoring to analyse use and inputs and confirm compliance.
• Report any serious incident which constitutes a breach to the relevant Market Surveillance Authority within 15 days.
• In the event of non-conformity, take corrective action to bring it into conformity or withdraw or recall it.
• Where there is no importer within the EU, the producer shall appoint an authorised representative within the EU to cooperate with national competent authorities.

In addition, recent debates seem to have introduced a requirement for bodies providing public services (such as healthcare or education) to conduct a “fundamental right impact assessment” before deploying high-risk AI systems.

• Ensuring they are designed to inform any natural person using it that they are using an AI system.
• In the event they generate or manipulate image/audio content to create “deepfakes”, disclose the artificial generation/manipulation of the content.

• That do not materially influence the outcome of decision-making but performing a narrow procedural task, for example AI model that transforms unstructured data into structured data or classifies incoming documents into categories.
• That review a previously completed human activity. I.e. merely providing an additional layer to human activity.
• Intended to detect decision-making patterns or deviations from prior decision-making patterns to flag potential inconsistencies or anomalies, for instance, the grading pattern of a teacher.
• Used to perform preparatory tasks only, to an assessment relevant to a critical use case. Examples include file-handling software.