Zhiyuan Jin, Praveer Tiwari, Geonho Roh,
Who is responsible when AI makes a mistake? This question is interesting to us given the growing use of AI in military operations. [See our essay on the topic here.] AI is often used to identify targets and process information much faster than humans, but it also makes us wonder who should be held responsible when something goes wrong. Is it the programmer who developed the system, the commander who approved its use, or the person in-charge who made the final decision? We have put together a collection of texts that explore these questions.
We begin with Robert Sparrow’s Killer Robots (2007), which asks who should be held responsible if an autonomous weapon commits a war crime. Is it the programmer, the commander who deployed it, or the machine itself? Sparrow examines these possibilities and argues that autonomous weapons can create a responsibility gap in which nobody can fairly be held accountable. His essay raises a troubling question: should we allow machines to make life-and-death decisions if we cannot determine who is responsible for their actions?
While Sparrow explores the possibility that nobody can be held responsible, Madeleine Clare Elish’s Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (2019) considers almost the opposite problem. Elish explains how people with limited control over automated systems can still receive the blame when those systems fail. Her concept of a “moral crumple zone” challenges the idea that simply keeping a human involved guarantees accountability. Even outside warfare, when an employee uses AI and something goes wrong, are they truly responsible, or are they simply the easiest person to blame?
These questions are complicated in the film Eye in the Sky (2016). The film follows a military drone operation in which the possibility of civilian casualties creates a difficult decision for military officers, political officials, and drone operators. Unlike the autonomous weapons discussed by Sparrow, the decisions here are in human hands. Yet responsibility is distributed across several people, each with different information and authority. If accountability is already complicated when humans make the decisions, what of when AI is part of that process?
Paul Scharre’s Army of None: Autonomous Weapons and the Future of War (2018) takes a broader look at this problem. Scharre examines why militaries are developing autonomous weapons, the advantages these technologies offer, and the ethical questions surrounding their use. Rather than treating automation as a simple choice between humans and machines, the book encourages us to think about which tasks can be delegated and which decisions should remain under human control. This brings us back to an important distinction: delegating a task does not necessarily mean delegating responsibility.
Finally, the International Committee of the Red Cross’s ICRC Position on Autonomous Weapon Systems (2021) considers what meaningful human control should look like in practice. The ICRC expresses concerns about losing human control over the use of force and recommends rules governing autonomous weapons. While the previous works explore the ethical and practical difficulties of assigning responsibility, this statement looks at possible solutions, including restrictions on targets, operating conditions, and human supervision.
Together, these works reveal a contradiction in how we think about automation. We increasingly delegate decisions to machines while continuing to expect humans to answer for the consequences. But responsibility and control are not necessarily the same thing. Perhaps the important question is not simply whether a human is involved, but whether that person has enough information, authority, and time to make a meaningful decision. Otherwise, keeping a human in the loop may preserve the appearance of accountability without guaranteeing it.
