What I Learned Today

Where does California’s Proposition 8 stand today?

This is the proposition which in late 2008 amended the state constitution to assert that “only marriage between a man and a woman is valid or recognized in California.” Today I was lucky enough to attend a talk by James Gilliam, the Deputy Executive Director for ACLU of Southern California. He gave a lucid, fascinating, and inspiring account of the history and current situation.

California is unique because it is the only state in which same-sex couples *did* have the right to marry (albeit briefly) before it was taken away. The question now is whether voters legally have the ability to take away that right, once established.

Last month, Judge Vaughn R. Walker issued a ruling that Proposition 8 violates the U.S. Constitution’s 14th amendment (equal protection) and therefore cannot stand. The ruling makes for some fascinating reading — it’s not as dry as you think it is. Effectively, the ruling says that the state of California has no interest (compelling need) to discriminate between same-sex and opposite-sex marriages, that Proposition 8 ultimately arises from a desire to impose one group’s moral view on another (which is not what laws are for), and that strict scrutiny (the highest level of skepticism) should be applied to any proposed law that would discriminate on the basis of sexual orientation.

From the ruling:

“Moral disapproval alone is an improper basis on which to deny rights to gay men and lesbians. The evidence shows conclusively that Proposition 8 enacts, without reason, a private moral view that same-sex couples are inferior to opposite-sex couples.”

This is just one of the many justifications that Judge Walker provided for his decision. In contrast, Prop 8 proponents have argued that it doesn’t infringe on any rights because gays can still legally marry… they just have to marry someone of the opposite sex. Sophistry at its best!

Proponents of Prop 8 have filed an appeal with the 9th Circuit Court. It turns out that in an appeal, the superior court does not review the facts and testimony to reach an independent decision; instead, the court focuses solely on whether the preceding judge misapplied the law. Another interesting aspect of this appeal is that it may not actually happen, because there may not be anyone with proper legal standing to bring the appeal. “Standing” is given only to those who are named in the original suit, with some exceptions. The defendants of Prop 8 named in the suit include Governor Schwarzenegger and Attorney General Brown, both of whom have publicly stated that they want Prop 8 to go away, and that they refuse to take on the appeal. (They are automatically named as defendants because in their positions they are charged with enforcement of the laws.) The Prop 8 folks who defended it before Judge Walker are permitted to bring the original suit, but likely not to appeal, because they are not named (they stepped in to represent that side because the government declined) and they have experienced no “impairment” caused by Judge Walker’s decision. We’ll find out in December whether the 9th Circuit Court thinks there’s anyone with standing who’s willing to argue for Prop 8. If not, Judge Walker’s decision will remain as is.

Of course, this doesn’t mean the issue will go away. California’s proposition system permits its voters to keep putting the same issue on the ballot, year after year. So even though Prop 8 has been declared unconstitutional, someone can write “Prop 8.1”, get enough signatures, and put it back on the ballot to revise the constitution all over again. If it passes, I guess that means it has to cycle through the courts again. I’m just appalled at this wasteful nonsense, and I hope that in reality this wouldn’t actually be permitted. Surely there’s some additional check against abuse of the proposition system? Please save California from itself!

I recently came across an article on the use of a neural network to predict which death row inmates would be executed and which would not. The authors of “An Artificial Intelligence System Suggests Arbitrariness of Death Penalty” argued that because they were able to train a neural network to successfully predict execution decisions using only irrelevant variables, then the (human) decisions being made must be arbitrary. Confused yet? Although their neural network achieved 93% accuracy, they argue that because information about DNA testing and the quality of each defendant’s legal representation was omitted, this performance is concerning. In their words,

“What we have demonstrated here is that ANN technology can predict death penalty outcomes at better than 90%. From a practical point of view this is impressive. However, given that the variables employed in the study have no direct bearing on the judicial process raises series questions concerning the fairness of the justice system.”

That is, the neural network must have identified a useful predictive pattern in the data, but in a sense it was “not supposed to,” so a pattern may exist where one should not be.

There are several problem with the arguments in and conclusion of this paper.

First, I don’t think the authors interpreted their result correctly. “Arbitrariness” was not at all demonstrated (despite the paper title). The neural network identified some sort of pattern in the data set that allowed it to successfully predict the outcome for 93% of previously unseen inmates. If they were executed “arbitrarily” (i.e., a random decision was made for each inmate), then the neural network would not have been able to learn a successful predictor. Instead, if the features really are irrelevant to the judicial process (they include sex, race, etc.), then high performance of the neural network instead shows bias in the system. There is some sort of predictive signal even in features that shouldn’t directly affect execution decisions.

Second, I’m not convinced that the features really are irrelevant. While sex, race, month of sentencing, etc., should (presumably) not be deciding factors in who gets executed, “type of capital offense” sounds quite relevant to me. If the neural network placed a heavy weight on that feature, I would be much less concerned than if it placed a high weight on “sex”. What was the neural network’s performance if the capital offense features were omitted? In fact, it would be interesting to use a machine learning feature selection method to pick out the “most useful” features from the 17 used in this study, to help identify any bias present.

Finally, the evaluation was quite limited, so our confidence in the conclusions should also be limited. The authors trained a single neural network on a single training set and evaluated it on a single test set. More typical methodology would be to use cross-validation, splitting the data set into, say, 10 test sets and, for each one, training a network on the remaining 9. This yields a much better estimate of generalization performance. Also, what about other machine learning methods? Is 93% achieved only by a neural network? What about a support vector machine? (SVMs have been shown to out-perform neural networks on a variety of problems.) What about a decision tree, which would yield direct insight into the decisions being made by the learned model? For that matter, what about neural networks with other network structures? Why was a network with a single hidden layer of five nodes used? Was that the only one that worked?

Naturally, my critique comes from a machine learning perspective. I have no legal training. I would be very interested in any insights or opinions on this work from those who do have a legal background. What is the value of this kind of study to the field? Is this an important subject to investigate? How could the results be used to positive benefit? What other questions were left unanswered by the authors of this paper?