A general-purpose language model produces the most plausible next word. That is a reasonable way to draft a paragraph. It is an unsound way to price a hire, size a pay gap or estimate a year of rework, because a board cannot audit a sentence that merely sounded right.
The usual answer is "keep a human in the loop". It is a good instinct, and the evidence says it is not enough on its own.
What the law now asks of you
In South Africa, section 71 of the Protection of Personal Information Act addresses automated decision-making. In outline, a person may not be subject to a decision that has legal consequences for them, or that substantially affects them, if it rests solely on automated processing intended to profile them, for example their performance at work, creditworthiness, reliability or conduct. There are exceptions, including where the decision is taken in connection with concluding or performing a contract and appropriate measures protect the person's legitimate interests.1
Two features are worth knowing. The word "solely" does a great deal of work: it invites arguments about how much human involvement is enough, and commentators warn that merely nominal oversight can hollow out the protection the section is meant to give.2 And POPIA contains no express right to an explanation, which leaves open how a person can meaningfully contest an automated decision.1 We are not lawyers and this is not legal advice. Take it to one before you rely on it.
For organisations selling or operating into Europe, Article 14 of the EU AI Act goes further for high-risk systems. They must be designed so that natural persons can effectively oversee them, and the overseers must be able to understand the system's capacities and limits, stay aware of automation bias, interpret its output correctly and override or stop it.3
Why a human in the loop is not a safeguard by itself
Parasuraman and Manzey's review of the research on complacency and automation bias is the standard reference. It finds that when an automated aid is imperfect, people make errors of omission and commission: they miss problems the aid did not flag, and they follow its wrong advice. It occurs in experts as well as novices, and it is not prevented by training or instructions alone.4
Ben Green's survey of policies that require human oversight of government algorithms makes the uncomfortable point directly. The policies assume that people can oversee algorithmic decisions effectively, and the evidence indicates they struggle to. Worse, the requirement can give agencies and vendors cover to deploy flawed tools. Green argues for a shift towards institutional oversight, in which the algorithm and the oversight design are justified with evidence before adoption.5
Where we draw the line
Our answer is to make the line structural rather than procedural. In our work, machines calculate, retrieve, propose and record. People frame the question, weigh what matters, decide, and answer for the decision. Concretely:
- No model sets a weight, asserts a constraint or issues a decision.
- Every figure the platforms show comes from deterministic code reading registered sources and evidence. A model may explain what that code has already shown. It never decides what is true.
- Every machine contribution is logged as a proposal that a named person accepted or rejected, so the record shows who decided and on what.
- Decisions are not scored against people. We show conditions and consequences, not verdicts.
There is a reason beyond risk. When a machine decides, nobody owns the decision, and a decision nobody owns is one nobody learns from. The organisations that will do well with AI are not those that automate the most judgement. They are the ones that are clearest about which judgements belong to people, and that make the machine show its working every time.
There is also a promise in it. For most of history, being able to see your options, cost them honestly and choose on purpose was rationed to those who could afford teams of analysts. Calculated, versioned and open to challenge, that clarity can be ordinary.