Why The Real Legal Risk Of AI Is Not Wrong Answers, But Invisible Decision-Making

Major legal discussions surrounding AI focus on the incorrect or biased answers it provides. Whether it echoes popular stereotypes about women or confidently agrees with both sides of a problem, discussions aim to address the implications of these answers. However, these discussions remain surface-level and miss the heart of the problem.

The real problem arises when we look behind the curtains of its decision-making mechanisms. As the Harvard Journal on Law and Technology describes it, this is the “black box problem.” Simply put, it refers to the fact that AI has invisible decision-making. We know what the user inputs and what the response is, but we have no idea how AI reaches that conclusion or what its reasoning was.

This is where the legal complications start. What is effectively happening is that the intention behind AI becomes impossible to make out. We may know that a response was biased or wrong, but we cannot determine whether the makers intended it that way. Even if AI is trained on the best scholarly works, it can still pick up on human biases that are inherently present within them.

The decision-making of AI has grown so complex that, forget courts, not even its makers fully understand how it reaches a conclusion. This leads to the next problem: evidence analysis. The code and data underlying AI systems are often treated as trade secrets, meaning lawyers representing plaintiffs must navigate numerous procedural hurdles to obtain them. Even after gaining access, it is nearly impossible to interpret this material, let alone establish concrete evidence before a court.

Assigning blame is another frequently discussed issue when considering the legal implications of AI. This, too, stems from its opaque decision-making. Since we do not know what triggered the faulty response, it becomes difficult to decide who should be held responsible. The error may arise from biased data, faulty instructions from engineers, or even the manner in which the user prompted the system, leading to widespread confusion.

The decision may be wrong; that is fairly certain. Legal complications arise because uncertainty lies at the heart of AI decision-making. Courts are confronted with situations where a wrong has occurred, but the process behind it remains unknown. At the same time, they are expected to apply principles such as audi alteram partem, even though it is impossible to identify or hear the “other side” of an invisible decision-making process.

Unless this invisibility is addressed, the use of AI risks weakening procedural fairness rather than strengthening legal decision-making.