Can neural networks predict the death penalty?

I recently came across an article on the use of a neural network to predict which death row inmates would be executed and which would not. The authors of “An Artificial Intelligence System Suggests Arbitrariness of Death Penalty” argued that because they were able to train a neural network to successfully predict execution decisions using only irrelevant variables, then the (human) decisions being made must be arbitrary. Confused yet? Although their neural network achieved 93% accuracy, they argue that because information about DNA testing and the quality of each defendant’s legal representation was omitted, this performance is concerning. In their words,

“What we have demonstrated here is that ANN technology can predict death penalty outcomes at better than 90%. From a practical point of view this is impressive. However, given that the variables employed in the study have no direct bearing on the judicial process raises series questions concerning the fairness of the justice system.”

That is, the neural network must have identified a useful predictive pattern in the data, but in a sense it was “not supposed to,” so a pattern may exist where one should not be.

There are several problem with the arguments in and conclusion of this paper.

First, I don’t think the authors interpreted their result correctly. “Arbitrariness” was not at all demonstrated (despite the paper title). The neural network identified some sort of pattern in the data set that allowed it to successfully predict the outcome for 93% of previously unseen inmates. If they were executed “arbitrarily” (i.e., a random decision was made for each inmate), then the neural network would not have been able to learn a successful predictor. Instead, if the features really are irrelevant to the judicial process (they include sex, race, etc.), then high performance of the neural network instead shows bias in the system. There is some sort of predictive signal even in features that shouldn’t directly affect execution decisions.

Second, I’m not convinced that the features really are irrelevant. While sex, race, month of sentencing, etc., should (presumably) not be deciding factors in who gets executed, “type of capital offense” sounds quite relevant to me. If the neural network placed a heavy weight on that feature, I would be much less concerned than if it placed a high weight on “sex”. What was the neural network’s performance if the capital offense features were omitted? In fact, it would be interesting to use a machine learning feature selection method to pick out the “most useful” features from the 17 used in this study, to help identify any bias present.

Finally, the evaluation was quite limited, so our confidence in the conclusions should also be limited. The authors trained a single neural network on a single training set and evaluated it on a single test set. More typical methodology would be to use cross-validation, splitting the data set into, say, 10 test sets and, for each one, training a network on the remaining 9. This yields a much better estimate of generalization performance. Also, what about other machine learning methods? Is 93% achieved only by a neural network? What about a support vector machine? (SVMs have been shown to out-perform neural networks on a variety of problems.) What about a decision tree, which would yield direct insight into the decisions being made by the learned model? For that matter, what about neural networks with other network structures? Why was a network with a single hidden layer of five nodes used? Was that the only one that worked?

Naturally, my critique comes from a machine learning perspective. I have no legal training. I would be very interested in any insights or opinions on this work from those who do have a legal background. What is the value of this kind of study to the field? Is this an important subject to investigate? How could the results be used to positive benefit? What other questions were left unanswered by the authors of this paper?