Bayes' Theorem in the Court of Appeal

Bernard Robertson and Tony Vignaux

In R v Adams the prosecution gave evidence of the results of a DNA test. The witness gave that evidence in the form of a likelihood ratio. The defence the produced Professor Donnely who explained to the jury how to combine that evidence with other evidence in the case (which all pointed away from the accused's guilt). The correct way to do this, of course, is by applying Bayes' Theorem. This was accepted by a Bench presided over by the Lord Chief Justice in R v Deen (CA, 21 December 1993).

Professor Donnelly then went further and explained to the jurors how they could use Bayes' Theorem to combine each of the other items of evidence in the case with each other. When the case went on appeal, the Court of Appeal (Rose LJ, Hidden and Buxton JJ) said that they had not heard argument about this and could not 'express a concluded view on the matter'. Nonetheless they had 'very grave doubt as to whether that evidence was properly admissible'.

The Court in Adams doubted the admissibility of the evidence because it seemed to trespass "on an area peculiarly and exclusively within the province of the jury, namely the way in which they evaluate the relationship between one piece of evidence and another". This raises some basic issues to which we shall return. Had the Court stopped at this point all might have been well. Unfortunately the reasons that their Lordships gave for this view were, it is respectfully submitted, the wrong ones.

The Court did admit that "Bayes Theorem may be an appropriate and useful tool for statisticians and other experts seeking to establish a mathematical assessment of probability". This statement is ambiguous. The word 'mathematical' here could refer simply to the appearance of precision, or it could imply a qualitative difference between 'mathematical probability' and other probability. We would reject the latter proposition. There is only one kind of uncertainty and there is only way of measuring uncertainty. That is called probability; the only logical way to reason in a state of uncertainty is in accordance with the axioms of probability.

The Court then made a number of detailed observations with which we deal in turn.

the theorem can only operate by giving to each separate piece of evidence a numerical percentage representing the ratio between the probability of circumstance A and the probability of circumstance B granted the existence of that evidence.

Two quibbles. First, probabilities may be expressed in percentages, but a likelihood ratio is just a number: it cannot be a percentage. Secondly, in the last part of this sentence their Lordships have transposed the conditional, the error discussed in R v Deen. A likelihood ratio is the ratio between the probability of the evidence given circumstance A and circumstance B. It describes the strength of the evidence in distinguishing between proposition A and proposition B. What their Lordships have described is the posterior odds that the Court is trying to assess. But the major point is that once these corrections are made this is a correct statement about Bayes' Theorem which is neutral in effect.

The percentages chosen are matters of judgement: that is inevitable. But the apparently objective numerical figures used in the theorem may conceal the element of judgement on which it entirely depends.

One would hope that these 'judgements' would be made in the same way that all assessments of probability should be made, rationally and by reference to the evidence. The figures given are merely expressions of strength of belief. Any system for expressing strength of belief must comply with some simple rules such as:

(a) if I believe that A is more likely to be true than B and that B is more likely to be true than C, I must believe that A is more likely to be true than A; and

(b) equivalent levels of belief are expressed equivalently and divergent levels of belief expressed divergently.

Not only are numbers a convenient way of achieving this, but it can be shown that any system of expressing strength of belief which complies with these rules can be reduced to numbers.

the theorem’s methodology requires, as we have described, that items of evidence be assessed separately according to their bearing on the accused’s guilt, before being combined in the overall formula

This is correct only up to a point. At some point in the deliberative process items of evidence have to examined separately, indeed it is hard to see how a body of evidence can be rationally assessed without some dissection. When an item of evidence is examined however, it must be considered in the light of the evidence already considered. This leads to the criticism of bayesian reasoning that the interdependencies of the items of evidence make the process far too complex. There are several answers to that. One is that all Bayes' Theorem does is make obvious complexities which exist in reality. It is hard to understand how decision making can be improved by deliberately ignoring them. Secondly, the idea that a jury sits in Court with a pre set 'prior probability', computes a likelihood ratio for each item of evidence and then combines them one by one is an unhelpful model. The jury considers the evidence when it has withdrawn. By that stage a large amount of evidence will have been accepted as true and the dispute will have narrowed down to a choice between two or three well-defined stories. At this point the jury's deliberation begins. The jury only has seriously to consider evidence which is genuinely in dispute and which distinguishes between disputed alternatives. This enormously reduces the problem of complexity.

That in our view is far too rigid an approach to evidence of the type that a jury characteristically has to assess, where the cogency of (for instance) identification evidence may have to be assessed, at least in part, in the light of the strength of the chain of evidence of which it forms part.

Where an item of evidence is genuinely part of a chain, it must be considered in that light and bayesian reasoning explains in formal terms how that is done. Clearly, an identification is affected by the Turnbull factors and these are to be considered when assessing the strength of the identification as evidence. If, however, the Court was referring here to the rule that a weak identification is treated differently according to whether or not there is supporting evidence, bayesian reasoning reveals this as illogical. The appropriate analogy when an identification and supporting evidence are considered is not a chain but a rope composed of several strands. The strength of one strand may affect the strength of the whole rope but it cannot affect the strength of another particular strand. The rule in Turnbull that a weak identification should only go to the jury if there is supporting evidence amounts only to saying that a weak identification can go to the jury if the case is otherwise strong and not if it is otherwise weak.

Jurors evaluate evidence and reach a conclusion not by means of a formula, mathematical or otherwise, but by the joint application of their individual common sense and knowledge of the world to the evidence before them.

But what does this mean?. Bayes' Theorem is merely the formal expression of how one applies common sense and knowledge of the world to the evidence. It makes clear whether evidence can be regarded as strengthening or weakening a case. While Bayes' Theorem is not the only tool provided by Bayesian reasoning, any analysis that does not comply with Bayesian principles is illogical and wrong. Few 'bayesian lawyers' argues that jurors should consciously apply Bayes' Theorem to each item of evidence as Professor Donnelly appears to have recommended. But whenever a Judge instructs a jury in how to consider evidence or an appeal Court rules on how facts are to be thought about, these instructions must be consistent with logic and reason, the formal expression of which is bayesian formulae. The obvious analogy is that the instructions that a parent gives when teaching a child how to ride a bicycle must conform to the laws of mechanics; but there is no need to have any conscious knowledge of mechanics before one can successfully teach a child to ride a bicycle.

Scientific evidence tendered as proof of a particular fact may establish that fact to an extent which, in any particular case, may vary between slight possibility and virtual certainty. For example, different blood spots on an accused’s clothing may, on testing, reveal a range of conclusions from ‘human blood’ via ‘possibly the victim’s blood’ to ‘highly likely to be the victim’s blood’.

Unfortunately there is a major misunderstanding here. For decades scientists have given evidence in this form, especially in paternity cases. The result is that the Courts have become used to hearing this kind of evidence, and become puzzled when the scientfic evidence appears to establish a fact to a virtual certainty but there is cogent non-scientific evidence pointing the other way. In fact there is no logical way of combining such a statement with the remainder of the evidence in the case, which is itself sufficient reason for rejecting it. The evidence in Adams was not given in this way, but lawyers have become so used to hearing this kind of statement that when evidence is correctly given, as it was in Adams, they hear it incorrectly, as evidenced by their Lordships' transposition of the conditional pointed out above.

In fact a scientist who expresses a conclusion of this sort usurps the role of the jury to a greater, and more insidious, extent than Professor Donnelly may have done. The assessment of a posterior probability, which a statement such as 'highly likely to be the victim’s blood’ is, requires the assessment of a prior probability. In paternity cases this has arbitrarily been treated as 0.5, but there is no warrant for this. The prior probability depends upon the other evidence in the case and is a matter for the jury. Professor Donnelly's sin was to make his assumptions and reasoning transparent, whereas the Court appears illogically content with expressions which conceal the expert's assumptions.

Individual jurors might differ greatly not only according to how cogent they found a particular piece of evidence (which would be a matter for discussion and debate between the jury as a whole), but also on the question of what percentage figure for probability should be placed on that evidence.

Again, the reference here should be to a likelihood ratio and not to a 'percentage figure for probability'. That apart, this statement appears to be tautological. Two people who disagree on the strength of a item of evidence will naturally disagree on its likelihood ratio. An important point however, is that the process of constructing a likelihood ratio will make clear why two people disagree. At the end of an argument structured in this way one or both may wish to change their view of the evidence.

different jurors might well wish to select different numerical figures even when they were broadly agreed on the weight of the evidence in question.

This would, of course, make perfect sense if means is that jurors only agree broadly and not precisely about the strength of the evidence. If two people agree that a piece of evidence is 'very strong' they must both presumably accord it a likelihood ratio higher than a piece of evidence they are agreed is only 'strong' and so forth.

They could, presumably, only resolve any such difference by taking an average, which would truly reflect neither party’s view; and this point leaves aside the even greater difficulty of how twelve jurors, applying Bayes as a single jury, are to reconcile, under the mathematics of that formula, differing individual view about the cogency of particular pieces of evidence.

At this point, it is respectfully submitted, their Lordships are wrong in law. The only matter on which the jury is required to be unanimous is that "the the prosecution [has] prove[d] the charge it makes beyond reasonable doubt." Mancini v. Director of Public Prosecutions [1942] AC 1 (HL) per Viscount Simon LC at 11.

The jury are not required to be unanimous as to how the offence was committed. This was regarded as 'clear beyond argument' by the Court of Appeal in Attorney General's Reference (No 4 of 1980) [1981] 2 All ER 617. A more dramatic example is the Supreme Court of Canada's upholding the verdict in Thatcher v The Queen (1987) 39 DLR (4th) 275 in which the prosecution told two mutually inconsistent stories.

Likewise the jurors are not required to agree on the evidential route by which they reach their individual verdicts. As Turner J in the New Zealand Court of Appeal put it in Thomas v. The Queen [1972] NZLR 34, at 41:

It is of course inherent in the process of conviction by jury that the jury must be convinced as a whole, and each member must be convinced individually, beyond reasonable doubt of the guilt of the accused. This necessarily extends to every essential element of the crime charge ... it does not logically follow that each of the members of the jury must base his or her individual conclusion upon the same reasoning as the others. Different members may individually be convinced beyond reasonable doubt of the guilt of the accused, by their individual acceptance of different facts. (emphasis in original)

A fortiori, where they reach their verdicts on the basis of the same facts there is no requirement for jurors to be precisely agreed on the strength of any particular item of evidence.

to introduce Bayes Theorem, or any similar method, into a criminal trial plunges the jury into inappropriate and unnecessary realms of theory and complexity deflecting them from their proper task.

It may indeed plunge the jury into unnecessary realms of theory and complexity, but it can hardly be described as a deflection from their proper task. Bayes' Theorem constitutes not a deflection from this task but a formalisation of it, a formalisation which, we would agree may not always be necessary or helpful to a juror, but which is the only appropriate yardstick for those, such as appeal Courts and academics, who judge Judges.

There is arguably a much simpler and more compelling reason why experts should not give evidence on how jurors should combine non-scientific evidence using Bayes' Theorem. If such evidence were admissible in Adams it would be admissible in every single criminal case. While this might be good for those qualified to give the evidence, this would not do anything for the cost-effectiveness of the criminal justice system.

In fact it is doubtful whether an explanation of Bayesian reasoning is 'evidence' at all. It is merely a detailed analysis of common sense. A Judge might in a particular case wish to hear a detailed explanation, but would then be hearing a witness in order to take judicial notice of a matter, not taking evidence. Such instruction could be included in counsel's closing addresses to the jury and in the Judge's summing up since Judges are required to instruct the jury on how to consider the evidence.

We have always favoured expert witnesses explaining the effect of the likelihood ratio by saying words to the effect of "whatever you consider the odds of guilt are on the basis of the other evidence, my evidence should cause you to multiply those odds X fold". This makes it clear that the witnesses are giving an opinion only on the strength of their own particular evidence and not of the case as a whole and also stresses that scientific evidence must be considered in combination with the other evidence in the case.

The ground for quashing the conviction was that the Judge's summing up concentrated on Bayes' Theorem

without indicating to the jury the more commonsense and basic ways in which it would have been open to them to weigh up the relative weight of the DNA evidence . . . [the jurors] were left by the summing up with no other sufficient guidance as to how to evaluate the prosecution case . . . in the light of the other non-DNA evidence in the case.

The defence case, however, was not that the DNA evidence should be weighed relative to the other evidence but that it should be combined with it. Apart from this point the Court of Appeal fail to explain what these 'more commonsense and basic ways' are. If it had attempted to explain it would have faced an insuperable difficulty. The guidance could either be meaningless waffle or it could have some content. Such content would either have to conform to the requirements of bayesian reasoning or be illogical and wrong.

We understand that Adams has now been retried in the Crown Court. Prosecution and defence agreed a method of explaining the application of Bayes' Theorem to the jury. We are told that the experts concerned may be publishing a paper explaining what occurred.

Bernard Robertson and Tony Vignaux are the authors of Interpreting Evidence: Evaluating Forensic Science in the Courtroom, published by John Wiley and Son Ltd (UK), 1995, where these matters will be found more fully discussed.