The Trolley Cart Problem is not an AI problem

Every time there is a discussion on the future of AI-powered Autonomous Vehicles, somebody put the Trolley Cart Problem (TCP) on the table. And every time this happens, I am annoyed. However, recently, I saw some mutual followers studying AI and Computer Science talking about how TCP is a fundamental problem for the future of AI and autonomous vehicles. So I think it is time to speak it loud: ‌the Trolley Cart Problem is not an AI problem!

The TCP just barely scratch the domain of the autonomous vehicle, and, for sure, it is totally not an issue for any foreseeable vehicle AI.

The Trolley Cart Problem

First of all, let’s look at the object of the discussion.

The Trolley Cart Problem is a thought experiment for an ethical problem proposed, in its earliest version by Philippa Foot in 1967. However, the current formulation comes from a 1976 philosophy paper by Judith Jarvis Thomson¹.

The problem, in its most straightforward variation, can be stated as follow:

There is a runaway trolley barreling down the railway tracks. Ahead, on the tracks, there are five people tied up and unable to move. The trolley is headed straight for them. You are standing some distance off in the train yard, next to a lever. If you pull this lever, the trolley will switch to a different set of tracks. However, you notice that there is one person on the side track. You have two options:

1. Do nothing and allow the trolley to kill the five people on the main track.
2. Pull the lever, diverting the trolley onto the side track where it will kill one person.

What is the right thing to do?

Many other variations exist: in some, we can find fewer or more people on the railway; in other, the people on the side-railway is known to be evil; or he is fat, old, tall, short, black, white, or with any other peculiarity you can imagine.²

Because the classical formulation talks about a vehicle and a decision to be made by the vehicle controller, it is natural to extend the ethical problem the TCP poses to the domain of AI and autonomous vehicles. However, as we will see, this extension is totally out of place.

The Trolley Cart Problem is an exploration tool

The first objection is that the TCP was never intended as a practical guiding tool for ethical decisions. Instead, the TCP is a tool for probing individuals’ subtle moral space (or better, a population’s average moral space).

Because it is hard to obtain valid ordering about ethical values (it would be weird to answer questions like “how many children lives is worth a fat man?”), you can use TCP for that. First, you ask several variations of the problem to a group of people (one in which to save 2 children, you need to steer the cart on an old man; one in which to save a woman, you need to steer the cart on a man; and so on and so forth). Then you compare the average responses, and you approximate the population’s moral choice function.

Therefore, if it is an exploration tool, there is nothing to “solve” in practical applications. Moreover, to apply all this to machines, we need to assume the existence of some hidden emergent moral authority to be probed. Unfortunately, using this assumption with the current state of AI is a bold claim at best.

The Trolley Cart Problem requires the AI to be a moral agent

The TCP is useful in investigating the agent’s ethics. Unfortunately, it is hard to believe the AI will have any investigable ethical authority for a long time.

There are two possible scenarios. In the first, the AI is developed “top-down,” therefore, the ethic is imposed on the system by its creators. Consequently, the ethics in question does not belong to the AI: the developers imprinted it on the machine, so it is the developers’ ethics we are questioning. Even a crowdsourced effort like the Moral Machine experiment has nothing to do with machine morality. The Moral Machine is an interesting exploration of human morality to obtain information to impose the human morality on the AI. But if we use this data to inject ethics into an AI, it will still remain human ethics. The ones that are morally responsible for the machine behavior are the ones who designed the “ethical subroutine” of the AI, not the AI itself.

In the second scenario, an ethical decision emerges from a low-level algorithm (e.g., Neural Networks and other statistical Machine Learning tools). In this case, though, emergence is insufficient to put the ethical burden of the machine’s actions on the machine itself. There are many statistical emergent behaviors in nature, yet we do not question their morality.

To have an ethic implies some form of self-awareness and the ability to deeply understand the consequence of the action on various levels.³ A machine does not currently possess self-awareness. I also do not believe that current Neural Network can go deep on the meaning of their action: they are complex optimization algorithms. They do not care about their outputs’ universal and societal implications unless they are clearly stated in their optimization goal.

The consequence of all this is that talking about “AI’s ethic” is nonsense. Again, we forget about the developer who decides the function the AI should optimize, the testers that are evaluating it, the political system that is making the laws to allow its execution among society. They are all these agents to carry the burden of ethical decisions at every development step. The AI is the mere executor.

The Trolley Cart Problem cannot be applied to most autonomous vehicles

And now, the most important reason why TCP is a bad problem for AI and autonomous vehicles: it cannot really happen in reality.

This may appear a weird statement. After all, I wrote before that extending a problem involving a train and a controller to autonomous vehicles is “natural”.

Unfortunately, a Trolley Cart Problem is based on two vital preconditions.

1. The collision must be imminent and unavoidable.
2. The agent in the situation must, nevertheless, have a deterministic choice over the distribution of the unavoidable harms.

In 99% of mundane road scenarios, these two conditions cannot be satisfied simultaneously. If a collision is imminent and unavoidable (because, for instance, the car lost control), then, well, the car lost control. Therefore it cannot control the harm distribution. If the car runs too fast to avoid a sudden child on the road, it is probably running too fast to take any other decision with the prescribed 100% accuracy.

Even the distribution of harm may be a problematic concept. Should a car steer down a bridge (killing their passengers for sure) to avoid an incident that may kill a couple of children? How can the car evaluate a complex variable scenario like that when the car itself is in a critical situation in the first place?

As soon as we relax TCP constraint, we fall into more practical and mundane scenarios that do not have the clean and deterministic information required by TCP. Consequently, any ethical decision coming up from an impossible scenario is just a fun thought experiment. Nothing practical can arise from that.

So why are we wasting time thinking about scenarios in which an autonomous car will never be involved?⁴

AI does not need an ethical answer (yet), they need a political one

The TCP is not a problem for autonomous vehicles because autonomous vehicles do not require an ethical answer: they require a political one.

For instance, autonomous vehicles will release drivers from moral choices and, therefore, from legal consequences. How will society decide how to allocate the consequences of an incident? To the manufacturers? To a control authority? How will we evaluate the profound modification that autonomous vehicles will bring to mobility, environmental impact, or urban space design? How do we human decide to take these challenges and harmonize them with the supreme requirement of safety?

There are these and much more urgent and exciting questions about the future of AI and mobility. Every time we think of the TCP as a serious problem, we are just wasting a lot of time.