Clear and Obvious Errors
How football's pursuit of perfect decisions made the game worse - and offers lessons for policymakers
On the outskirts of Monterrey, a mountain looms at one end of the BBVA stadium. The walk to the ground is a maze of suburban streets that, last week, were full of locals ecstatic about the World Cup. Some had bottles of tequila and shot glasses on tables, offered free to matchgoers; others offered a selection of colored envelopes containing handwritten Bible passages in both English and Spanish.
I made that journey last Sunday. That meant I was about thirty feet from the ecstatic face of Mattias Svanberg when he scored to put Sweden 4-1 up against Tunisia. I’m guessing that part of his astonishment came from the fact that he’d come on the field just 18 seconds before. That made it the second quickest goal ever scored by a substitute in the World Cup.
But then I saw his face fade to dismay as he clocked the raised flag of the referee’s assistant in front of me. Offside: the goal was wiped away. A period of confusion followed - no one inside the ground knew what was happening. Eventually, the big screen showed it was, in fact, a GOAL.
Svanberg wheeled away in celebration once again. It was a muted attempt to simulate the joy he’d felt five minutes before – but at least he had the goal and the record.
I only found out what happened the next day. The referee’s assistant hadn’t spotted that Alexander Isak had got the slightest touch, putting Svanberg onside. That wasn’t the assistant’s fault: no human could have seen that in real time. But video reviewers could. They used a new “snicko” feature, based on a sensor inside the ball, to catch the touch. I didn’t know this kind of thing existed outside of cricket.
I had seen, live, the ideal vision of video assisted refereeing (VAR): a crucial mistake by the official was caught using superior technology. But there’s a growing sense that this vision is proving to be false – as I’ve written before, there’s a widespread and growing sense that VAR, despite all its capabilities, is in deep crisis.
Why should you care if you’re not a football fan? Well, I believe that the story of VAR offers a broader warning for anyone trying to find technical solutions to fix problems – especially policy makers.
Simple technocracy can assume that it’s enough to just find better evidence or more sophisticated data. Instead, VAR suggests the worrying possibility that those advances just open up new ways that we can be wrong – and that problems get recreated at a different level. In other words: technology will never eliminate the need for human judgment, but rather recreate that need in a new form.
Good intentions and the grimpen mire
The basic idea behind VAR is that human officials can make mistakes, and technology can help them make correct decisions more often. And the truth is that referees do make mistakes: in the last decades, a series of big calls were later shown to be wrong. Most notoriously, a handball by Thierry Henry – not caught at the time – denied Ireland a place at the 2010 World Cup. The offside I saw in Monterrey is another example.
So, the policy comes from the admirable desire to improve consistency and fairness. Most policies have good intent! And in some cases, tech clearly has helped. Goal line technology is maybe the best example: for a goal to be scored, the whole ball must cross the line. That’s one of the biggest calls a referee can make, and it’s pretty much been solved by a “goal decision system”, which sends a notification when cameras show a goal was legitimate. Tennis has mostly replaced human line judges with a similar system.
The problems come when there’s a mix of technology and human judgment, when there are officials sat in a little room, watching video feeds and trying to interpret the meaning of what just happened on the field. When we start to edge beyond the original terrain of “clear and obvious errors”, our footing fails, and we start getting sucked down into the grimpen mire.
The handball and offside rules show this most clearly. If we can see even the tiniest touches of ball to hand, we need to work out which ones are fouls and which can be tolerated. If we can mark the position of players’ bodies with extreme precision, we need to define more precisely which body parts count for being offside.
We start coming up against the fact that there’s a messiness to reality that the precision of technology cannot resolve. I’m reminded of the fact that even the most powerful quantitative approaches rest, at their base, on qualitative judgments about categories (you can only count cats if you can agree what counts as a cat). As the journalist Daisy Christodoulou puts it in her book I Can’t Stop Thinking About VAR, “The problem isn’t that VAR is inaccurate. It is that it is too accurate… its accuracy is revealing things about reality that we don’t like.”
Technology cannot bridge the judgment gap entirely. I want to highlight two results that follow: errors become fractal, and the “solution” starts changing the game as a whole.
Fractal errors
The implicit logic of introducing slow-motion replays is that they will allow officials to work out what “really” happened. But that’s wrong: once you’re beyond the simple question of “did the ball cross the line?”, the need for interpretation means that what “really” happened becomes a question for our judgment. The same goes for many attempts to resolve policy questions through technical means. How far do formal assessments capture a good education? Technical interventions into human systems rarely eliminate human judgment, but rather relocate it – and potentially amplify it.
So, the officials staring at the screen are still trying to work out if one object is closer to a line than another object, just like the ones on the field – they’re merely doing it on a microscopic level. They’re still having to decide what counts as concepts like “intention” or “phases of play”, and they’re still making errors of judgment like drawing the measuring lines in the wrong place. In this way, technology seems to make errors fractal: reproducing them at smaller and smaller scales, rather than eliminating them.
In fact, the technical solution can create “brand-new mistakes, of a type we haven’t seen before.” There’s some evidence that slow-motion replays distort judgment rather than enhancing it – most notably when it comes to judging intentions. Introducing a whole new set of officials means creating a whole set of opportunities for communication errors. In one case, the official did not challenge a disallowed goal (in time) because they mistakenly thought the on-field official had given it.
VAR has made decisions more accurate overall. But the issue is that it raised the promise of total accuracy. Pre-VAR, a wrong call was a misfortune. Fans shouted, cursed, and moved on; errors were regrettable but human. In contrast, technology made perfection the reference point. The unavoidable fractal mistakes are now seen as a betrayal of that promise by “the system” itself. As Thomas Concannon of the Football Supporters’ Association said: “We discuss refereeing in more detail than we ever have before, even though we have something that’s supposed to make it even more accurate.”
Solutions creating new problems
Once officials started realizing how technology was uncovering the messy reality of on-field play, what did they do? They started changing (or “clarifying”) the rules to try to ensure VAR could cope. As a result, the definition of handball became more and more complicated. This did not make decisions easier, much as the growth of Standard Operating Procedures and handbooks can impede judgment. As one observer put it, “The attempt to define in prose every possible combination of circumstances in which a ball might hit a hand has not worked.”
Instead, what happened was that the new rules started changing the way the game was played, leading to bizarre situations like defenders running with their hands behind their backs. Attempts to enforce the rules “better” led to new rules and unexpected changes to the game these rules were meant to serve.
Many years ago, I spent some time trying to work out how policy could be made better. One problem, I realized, was the idea that you could clearly separate “a policy” from how it was implemented. I strongly believe that a policy is what is realized in practice - in contrast to the approach that says “it was a good policy but the implementation let it down”.
I think the example of VAR illustrates the point: the overall goal (or policy) was to increase fairness and consistency, but the actions to implement that goal were not mere technical details: they changed the nature of the game itself.
There’s a deeper challenge here as well. Technocratic policies or reforms often approach goals in a relatively narrow way: we need to reduce fraud, or boost crop yields, or increase cancer screening rates. Therefore, the best approach is to identify the relevant high-quality evidence and execute plans based on that evidence. Taken in this narrow way, VAR is a success, since decision accuracy has increased!
Yet, as I’ve written, the price of this result has been severe damage to the experience of watching football. The experience of watching football (flow, spontaneity, shared emotional moments) were apparently not considered when pursuing the goal of decision accuracy. I think many fans have realized that they don’t just care about fairness and accuracy, if they have to be traded against feelings of authenticity and enjoyment.
Narrow technocracy handles these kinds of value tradeoffs badly. Technocrats would, for example, push against a policy to reduce class sizes if the money could be used more effectively to increase attainment elsewhere. Yet, if this policy were the centerpiece of a newly-elected government and was wildly popular (as for Tony Blair’s government in 1997), is it quite right to say it is a bad policy? Similarly, is it wrong to wish for imperfect decisions yet a more thrilling, immediate game? A wider perspective allows judicious tradeoffs to be made.
Closing the judgment gap
Many people realize that any VAR solution needs to balance the tension between technical precision and individual judgment, between absolute consistency and “common sense”. Instead, “we have ended up with the worst of both worlds - lack of consistency and lack of common sense.”
I’ve already proposed several changes to improve the experience of VAR for fans, such as increasing transparency and introducing player challenges. So here I want to go up a level and consider what might help us handle the bigger tradeoffs that are needed.
If human judgment cannot be eliminated, then we need to be clearer about how it operates - and what kinds of judgment we’re talking about. I am wary of adopting any simple distinctions between the functions of the left and right brain hemispheres. But I think that Iain McGilchrist makes a convincing case that the hemispheres can be seen as representing two different approaches to the world.
One (left) focuses on isolating elements from their context, so they can be manipulated, analyzed, and discussed. This is much like the VAR officials sat in their box, taking one specific moment out of the game, zooming in, and playing it back and forth. The other (right) considers the picture as a whole, making decisions based on an overall sense of the situation that is nuanced, sensitive, and hard to articulate. This is like the referee on the field, in the middle of a noisy stadium, trying to make reasonable decisions.
Both are legitimate ways of making judgments, but we need to: a) understand how far they are compatible; b) decide which one we want to privilege if they are not.
These are not abstract questions; they go to the heart of policy decisions. The tension between the need for consistency and the need for context-specific discretion goes to the heart of the public sector. We want both to retain the principle of equal treatment and to have our circumstances taken into account. Individual judgment cannot be eliminated, but neither can it rule.
My solution here is the idea of “system stewardship”, which is all about understanding the complexity of the issue and system you’re dealing with. For some simpler problems (e.g. did the ball cross the line?), we can adopt technological solutions wholesale, just like automated line calls in tennis. For more complex situations (e.g., was this handball?), we have to acknowledge that not everything can be locked down in advance, and therefore the key is to set clear, resilient “rules of the game” that provide space for individual action.
The irony is that when I first proposed system stewardship, fifteen years ago, I used football as an example of how system stewardship worked well! The rise of VAR has changed my view. And yet the concept itself has only become more pressing - the danger is that those developing AI adopt a narrow technical view of success as technical achievement, and neglect a broader view of what we want from that technology.
With that in mind, this year I will be creating an updated version of the system stewardship approach, with AI as both a new tool and an urgent test case. In the meantime, I’ll keep watching the football.