The element the judges are worst at

The court that contradicts itself showed that a bench of judges can reach opposite verdicts on the same case depending on whether it votes on the elements of the legal test one at a time, or on each judge’s own verdict. Which vote finds the truth more often then asked which procedure is more often right, if each judge is independently right about each element with some chance $p$, and every premise makes the verdict vote stricter followed the comparison as the number of elements grows: the verdict vote needs judges right on each element more than $(1/2)^{1/k}$ of the time to find liability where it exists, while the premise vote needs only better than even.

All of that assumed one number $p$ for every element. That essay ended by naming the assumption as the place where the model is least like a court. Judges are usually more reliable on the law than on a contested fact; two elements may turn on the same disputed evidence, so that a judge who misreads it gets both wrong. This essay lets the competence differ from element to element and lets a judge’s errors be correlated, and computes both procedures exactly.

The model, with one number per element #

Each judge is right about element $j$ with chance $p_j$, now different for different $j$. Judges are independent of one another, as before. A judge’s behaviour on two elements is then one of four joint outcomes — right on both, right on the first only, right on the second only, wrong on both — and when a judge’s errors on the two are independent their chances are products such as $p_1 p_2$ and $p_1(1 - p_2)$. A court of $n$ judges is a draw of $n$ such outcomes, and the number of judges in each of the four categories follows a multinomial law.

The two procedures read the same draw differently. The premise vote finds liability when a majority accepts the first element and a majority accepts the second, which may be different majorities. The verdict vote finds liability when a majority accepts both, the same judges each time. Summing the multinomial over all the ways $n$ judges can split into the four categories gives each procedure’s exact chance of finding liability. For a court of eleven there are 364 such splits, and nothing needs to be simulated.

In the hero figure the second element is easy, judged correctly ninety-five times in a hundred, and the first is varied from a coin toss upwards. The dotted curve is the chance that a majority of eleven is right about the first element alone — what a court would achieve if that element were the whole case. The premise vote follows it almost exactly, since its majority on the easy element is nearly certain to be right. The verdict vote does worse at every competence: 68 per cent against 75 when the hard element is judged right six times in ten, 97 against 99 at eight in ten. A single weak element drags both procedures down to its own level, and the verdict vote pays an extra price for the strong element’s small doubt, because each judge’s doubts multiply before the court counts them.

Three judges and a hard fact, by hand #

The mechanism behind the hero figure fits on a bench of three. Suppose the case has two elements, a point of law and a disputed fact, and the defendant is in fact liable. All three judges get the law right. Two of them get the fact right and one gets it wrong.

Voting element by element, the court accepts the law unanimously and the fact by two votes to one, and finds liability — correctly. Voting on verdicts, two judges find liability and one does not, and the court again finds liability. With three judges and one hard element, one error on the fact costs nothing under either procedure. Now let one of the two judges who got the fact right also doubt the law. Element by element, the court still accepts the law by two to one and the fact by two to one, and still finds liability. On verdicts, only one judge — the one right on both — finds liability, and the court acquits. A single doubt on the easy element, held by a judge who was right about the hard one, is what tipped the verdict vote, and it could tip it only because that judge’s verdict needed both elements at once.

That is the product rule in miniature. The verdict vote asks how many judges are right about everything, and a judge’s chance of that is $p_1 p_2$, the product of the competences, which a small doubt on the easy element erodes as surely as a large doubt on the hard one. The premise vote asks two separate questions and needs each answered by a majority, so it is governed by the smaller of the two majorities’ chances, and a small doubt on the easy element is absorbed by the easy element’s large majority. With equal competence the product and the minimum are tied together. With uneven competence they come apart, and every figure in this essay is a picture of the gap between them.

Four worlds, and where each procedure fails #

The liable world is one of four. Either element may in fact be satisfied or not, and only when both are is the defendant liable. With uneven competence, which element fails matters.

The verdict vote’s weakness is where it always was: in the liable world, where every judge must be right about everything to vote for liability, it finds it 79 times in a hundred against the premise vote’s 85. Where the easy element fails, both procedures are certain to acquit. The new case is the third world, where the hard element fails and the easy one holds. The premise vote must find the failure by a majority on the element the judges are worst at, which it does 85 times in a hundred. The verdict vote does better, 88, because a judge who doubts either element votes against liability and the doubts on the easy element add to the doubts on the hard one in the defendant’s favour.

So uneven competence creates a world in which the verdict vote’s caution is right and the premise vote’s literalness is not, and it is precisely the world in which the dispute is about the element the judges find hardest. That is not an exotic case. Contested trials are contested because some element is hard to judge, and that element is the likeliest place for a case to fail.

The asymmetry also changes how the two kinds of error should be read. Under equal competence, the verdict vote’s errors were all of one kind — failing to find liability that exists — and four ways out, and what each costs could describe the procedure as cautious. With one hard element the premise vote acquires errors of the other kind: it finds liability that does not exist, and it does so precisely when the hard element is the one that fails. A procedure is not cautious or bold in general. It is cautious or bold about each element in proportion to how well the judges can see that element, and the hard element is where the premise vote is boldest.

A map of the comparison #

A single number for each procedure needs a weighting of the four worlds, and the earlier essays used the simplest: equal weights. That weighting puts three not-liable worlds against one liable, and it is the comparison the next figure draws for every pair of competences.

The premise vote wins over most of the square — 609 of the 676 cells — and by the widest margin where both competences are moderate, around seven or eight in ten, which is where the comparison with equal competence found its crossover. The verdict vote wins along two strips at the edges, where the judges are barely better than chance on one element. There the premise vote’s majority on that element is nearly a coin toss in every world, while the verdict vote at least has the other element’s doubts pushing towards acquittal, which the averaged comparison rewards three times in four.

The map does not say which procedure a court should use. It says the answer depends on where the court’s competence lies, and that the procedure most often recommended — vote on the premises — is the wrong one for a court that is close to guessing about one element, if acquitting the not-liable matters as much as convicting the liable. That conditional is the real content: as deciding the premises or the conclusion argued, the choice of procedure embodies a choice about which errors to fear.

The same competence, spread differently #

The verdict vote has a property that becomes striking once competence can vary. A judge’s own verdict in the liable world is right exactly when the judge is right on both elements, with chance $p_1 p_2$. So the verdict vote depends on the two competences only through their product.

The figure holds the product at 0.72 and moves from an even court, right 84.9 times in a hundred on each element, to a lopsided one, right 72.5 times on the first and almost always on the second. The verdict vote finds liability 94.2 per cent of the time throughout: to it, these are the same court. The premise vote finds liability 99.4 per cent of the time with the even court and 94.7 with the lopsided one — still better, but by a margin that has shrunk from five points to half a point.

The reason is that a product spread evenly puts both premise majorities well above a half, and the jury theorem amplifies each; spread unevenly, it puts one majority close to a half, where the amplification is weak, and the premise vote is only as good as its weakest majority. Uneven competence therefore erodes the premise vote’s advantage while leaving the verdict vote untouched. The more lopsided the judges’ abilities, the less the choice of procedure matters in the liable world, because both are governed by the element nobody can judge.

When the errors come together #

Independence of a judge’s errors on different elements is the assumption the earlier essays leaned on hardest, and it is the one most obviously false. Whether a duty existed and whether it was breached often turn on the same facts; a judge who believes the wrong witness misreads both. The next figure lets a judge’s right-or-wrong outcomes on two elements be correlated, with correlation $\rho$, while each element is still judged right eighty times in a hundred.

Correlation barely moves the premise vote, which counts each element separately and so sees only each element’s majority. It transforms the verdict vote. With independent errors a judge is right on both elements with chance $0.8^2 = 0.64$; with correlation $\rho$ the chance is $0.64 + 0.16\rho$, because errors that come together leave more judges right on both. In the liable world the verdict vote rises from 83 per cent to 99, and at $\rho = 1$, where every judge is right on both or wrong on both, the two procedures are the same: a judge’s verdict is then a single right-or-wrong call, and so is each premise vote.

Averaged over the four worlds, the verdict vote overtakes the premise vote at $\rho = 0.66$. A court whose members’ mistakes are strongly linked across elements does better, on this measure, by voting on the verdict — the conclusion of the equal-competence comparison reversed by an assumption that is more realistic than the one it replaced. A majority wiser than its members found correlation between judges to be the enemy of the jury theorem; correlation within a judge is, here, the friend of the verdict vote.

A larger court, one element hard #

Court size was the subject of the previous comparison’s sharpest finding, that the verdict vote can get worse as judges are added. With uneven competence the threshold for that to happen moves.

With competences 0.6 and 0.8, each judge’s own verdict in the liable world is right with chance 0.48, just below a half, so the jury theorem works against the verdict vote: 46 per cent for a court of five, 39 per cent for fifty-one. The premise vote’s weaker majority is a majority of judges right sixty times in a hundred, comfortably above a half, so it improves with size, to 93 per cent at fifty-one. The relevant threshold for the verdict vote is now on the product, $p_1 p_2 > 1/2$, which is a constraint on the geometric mean of the competences; for the premise vote it is on the smallest, $\min p_j > 1/2$. With equal competence those were $(1/2)^{1/k}$ and $1/2$, and the difference between them was the whole of the earlier essay. With uneven competence the premise vote’s condition is about the weakest element and the verdict vote’s about the average, and a court can satisfy one without the other in either direction.

Why the procedures cannot be mixed to fix it #

Since each procedure wins in different worlds, a natural thought is to combine them: find liability only when both procedures agree, or when either does. The first is stricter than either and inherits the verdict vote’s weakness in the liable world; the second is laxer and inherits the premise vote’s weakness in the world where only the hard element fails. Any rule that looks only at the two procedures’ outputs trades one error for the other, which is the shape of the impossibility no rule escapes the doctrinal paradox proved for consistency rather than accuracy.

What would help is information the procedures discard: how sure each judge is, element by element. A court that weighted each judge’s vote on an element by that judge’s competence on it — the rule that maximises the chance of a right majority when competences are known, with weights proportional to $\log\big(p/(1 - p)\big)$ — would give the hard element’s votes less weight than the easy element’s in each judge’s verdict. But competences are not known, and estimating them from a court’s record requires knowing which of its past verdicts were right, which is what the court was convened to determine. The nearest consistent verdict looked for a principled middle way through distance rather than probability, and it faces the same limit: without knowing where the court is likely to be wrong, no rule can know which of its majorities to trust least.

What the exact sums assume #

Every number here is an exact sum over the multinomial, so nothing is estimated; what can be questioned is the model. Judges are independent of one another, which the correlation figure relaxes only within a judge. Each judge is equally competent, which a real bench is not: a majority wiser than its members showed that unequal judges call for weighted votes, and in a court with unequal judges and unequal elements the two inequalities interact in ways these figures do not trace. The four worlds are weighted equally in the averaged comparison, which means a quarter of cases are liable; a court whose docket is mostly meritless claims, or mostly meritorious ones, weights them differently, and the map would redraw its boundary.

And the model has no deliberation. Judges vote as they would alone. A court that discusses the elements before voting may move its members’ errors together — increasing the correlation that, here, helps the verdict vote — or apart.

Still open: what deliberation does to the errors #

The correlation figure suggests a sharper form of the question the earlier essay left open. If deliberation tends to make each judge’s errors on different elements more consistent with one another — a judge persuaded about the facts is persuaded about every element that turns on them — then deliberation pushes a court towards the regime where the verdict vote is as good as the premise vote, and the doctrinal paradox becomes rarer because judges’ views become more internally coherent. If deliberation instead makes judges’ errors on the same element more alike across the bench — everyone persuaded by the same advocate — it attacks the independence both procedures need. Which effect dominates is an empirical question about how judges update, and models of deliberation give different answers depending on how persuasion is assumed to work. The figures here say only that the answer matters, and in which direction each effect would move the comparison.