By Claire Marzo, on the basis of a keynote speech delivered by Nicola Countouris, UCL.
The lecture was presented on 13 March 2026, as part of the PROSPER lectures series. It was also part of a one-day symposium entitled “Human at the Helm: Work and Artificial Intelligence” (WAI@U project (AURORA network), IAG4UPEC project, an ERASME (ANR), EU Action COST P-WILL, Maison de l’lle de France, cité internationale universitaire de Paris.
Nicola Countouris reflects on the proposal to put the “Human in Command” when it comes to work in an era of artificial intelligence and algorithmic management and delivers an EU Law perspective.
- European Union Law Perspective
The principle of “human-in-control” has emerged as a central concept in European Union regulation of artificial intelligence, both generally and within the specific context of employment. Several key legal instruments articulate and operationalise this principle.
Under the General Data Protection Regulation (GDPR), Article 22 establishes the right not to be subject to solely automated decision-making. It further provides individuals with the right to obtain human intervention and to contest decisions affecting them. This provision reflects an early recognition of the need to preserve human agency in algorithmic governance.
The EU Artificial Intelligence Act develops this approach further. The AI Act does not treat human control as the sole governing principle, as it explicitly prohibits certain uses of AI deemed to present “unacceptable risks.” Alongside this prohibition/precautionary framework, the Act also deploys a human oversight approach, particularly for “high-risk” systems.
In the employment context, many AI systems fall within the “high-risk” category, notably those referenced in Annex III, including systems intended for use in workplace management and decision-making. For such systems, Article 14 imposes a human oversight obligation. This includes requirements that AI systems: incorporate interfaces that are understandable to human operators; allow effective intervention, including the possibility to halt system operation (“stop button”); and ensure that human overseers possess the necessary competence and capacity to supervise the system effectively.
Furthermore, human controllers must be able to interpret algorithmic outputs and identify potential biases. In particularly sensitive domains, such as biometric identification, additional safeguards may apply, including multi-person human verification.
However, Article 6(3) introduces a significant derogation: systems may be exempt from high-risk classification—and thus from human oversight obligations—if they are deemed not to materially influence decision-making. This creates a potential loophole, as many workplace AI systems could escape stricter regulation despite having meaningful practical effects.
Beyond the AI Act, the Platform Work Directive reinforces the importance of human oversight in labour relations. Articles 10 and 11 are particularly relevant. Article 10 requires the allocation of sufficient human resources to ensure meaningful review by competent and trained personnel, including the ability to override automated decisions. Article 11 grants workers the right to request review and rectification of decisions affecting them.
These protections may, in practice, be more directly relevant to workers than those contained in the AI Act. However, their scope is currently limited to platform workers, raising the question of why similar safeguards should not apply universally across all forms of employment. Recent proposals by the European Parliament seek to extend such protections more broadly, including through enhanced trade union rights and collectivised guarantees of meaningful human review.
- The Problem of Control
The concept of “control” is fundamental in labour law. It is traditionally used to determine the existence of an employment relationship, often framed in continental legal systems as a relationship of subordination.
In the context of human–machine interaction, however, the notion of control becomes problematic. Power asymmetries between workers and employers are already significant; the introduction of AI systems may further exacerbate these dynamics. Humans supposedly in control of AI will also be employees subordinated to the employer, who also controls, in the sense of owning/deploying, the AI systems. This is an uncomfortable triangle. Even where workers are formally empowered to review or override automated decisions, practical and psychological barriers may limit the effectiveness of such rights.
Workers may find it difficult to challenge AI outputs that appear more “objective,” data-driven, or intelligent than human judgment. This can lead to a phenomenon of psychological submission, of acquiescence to AI decisions, or “automation bias,” whereby individuals defer to algorithmic decisions rather than critically assessing them. Article 14(4)(b) of the AI Act explicitly warns against such risks, noting the danger of over-reliance and acquiescence.
This dynamic is reinforced by broader structural pressures, including productivity demands and managerial expectations. Workers may internalise the authority of AI systems, gradually losing their capacity for independent judgment. Over time, this may result in deskilling, reduced professional autonomy, and diminished academic or intellectual freedom.
Concrete examples illustrate these concerns. In the legal profession, instances have emerged in which lawyers relied on AI-generated case law containing fabricated authorities, raising serious issues regarding professional standards and duties of care. Courts have begun to respond by warning of sanctions, including threats of contempt of court, in cases of uncritical reliance on AI outputs.
More broadly, there is a risk that human reasoning itself may be shaped and constrained by machine-generated knowledge, leading to a gradual erosion of human agency.
III. The Role of Courts and the Limits of Human Oversight
It is important to acknowledge that human decision-making is not infallible. The introduction of human review mechanisms does not automatically guarantee fairness or accuracy.
For example, litigation involving the Uber Eats platform has highlighted the limitations of both automated and human oversight systems. In one case, a courier was denied access to the platform after failing a facial recognition check. The worker alleged that the system was biased against non-white drivers. Although the company attributed the issue to human error in the review process, the case underscored that neither automated systems nor human intervention alone can fully prevent discriminatory outcomes. The dispute was ultimately settled, leaving unresolved questions about accountability.
This example demonstrates that human review is neither a complete defence nor a panacea. Rather, it must be understood as one safeguard among many, whose effectiveness depends on institutional design, training, and broader power structures.
Conclusion
The human-in-control principle occupies a central place in contemporary EU regulation of artificial intelligence, particularly in the field of employment. While it offers important safeguards—such as the right to review, contest, and override automated decisions—it also faces significant practical and conceptual limitations.
Human oversight may be undermined by power imbalances, psychological biases, and structural pressures within the workplace. Moreover, legal loopholes and limited scope—particularly in the distinction between high-risk and non-high-risk systems—risk weakening its effectiveness.
Accordingly, reliance on the human-in-control principle should be approached with caution. While certain decisions—especially those involving management, discipline, or matters requiring empathy and emotional intelligence—should remain within human competence, this presupposes that AI systems are appropriately limited in scope.
Ultimately, meaningful protection of workers’ rights may require not only human oversight, but also stronger collective mechanisms, including trade union involvement, and a broader rethinking of the role of AI in labour relations.
