Forensic scientists, individually and as a group, want to be completely logical, open and transparent in their approach to the evaluation of evidence. Such an assertion is unquestionable. Further, I am sure that most document examiners believe this is exactly what they are achieving when they apply the procedures outlined in various traditional textbooks or the SWGDOC/ ASTM standards; for example, the SWGDOC Standard for Examination of Handwritten Items. Given the very understandable desire to be logical, I find it strange that so many people have a negative attitude towards anything and everything “Bayesian” in nature. After all, a logical approach to evidence evaluation that conforms to the overall Bayesian philosophy or approach is, quite literally, the embodiment of logic (more specifically, probabilistic logic).

Expert witnesses serve an important role in any justice system.  The Criminal Code of Canada, for example, provides for such testimony in section 657.3 (1):

In any proceedings, the evidence of a person as an expert may be given by means of a report accompanied by the affidavit or solemn declaration of the person, setting out, in particular, the qualifications of the person as an expert if (a) the court recognizes that person as an expert and (b) the party intending to produce the report in evidence has, before the proceeding, given to the other party a copy of the affidavit or solemn declaration and the report and reasonable notice of the intention to produce it in evidence.

Unlike lay witnesses, expert witnesses are permitted and expected to express their opinion and give testimony about their belief relating to the matter at hand based upon special knowledge or ability that they possess. In 1931, the Supreme Court of Canada, in Kelliher (Village of) v. Smith, [1931] SCR 672, quoted Beven on Negligence, 4th Ed., and wrote:

To justify the admission of expert testimony two elements must coexist:
(1) The subject-matter of the inquiry must be such that ordinary people are unlikely to form a correct judgment about it, if unassisted by persons with special knowledge.

(2) The witness offering expert evidence must have gained his special knowledge by a course of study or previous habit which secures his habitual familiarity with the matter in hand.

The topic was discussed further and expanded upon greatly in the seminal ruling R. v. Mohan, [1994] 2 S.C.R. 9

Ultimately, the first responsibility of any forensic expert (expert witness) is to evaluate evidence which falls within their scope of reference, in terms of particular and specific propositions of interest in a given case, while taking into account relevant background information. Then the expert must convey the meaning of that evaluation, as it applies to the matter at hand, to an interested party. Very often that will be initially an investigator or legal counsel. Sometimes it will proceed to a judge or jury in a court of law.

It is critical to remember the obvious point that, as a forensic expert our evidence is part, but only part, of the overall process whether it be a trial or still an investigation. In either situation, the decision to proceed one way or the other rests with that other party and generally involves a variety of different evidence types, including our own. That’s just another way of saying that all evidence should be considered by the trier-of-fact when reaching their final decision on disposition of the matter. And, as experts, we rarely have access to anything beyond our little piece of the puzzle. For obvious reasons, that is a good thing.


As such, it is my opinion that the focus of the expert’s work should be the determination and expression of a ‘likelihood ratio’ (or LR)1 that relates to the support provided by the evidence for each of at least two competing propositions, representing (as much as possible) the positions of the parties in a legal dispute. The LR is only one part of Bayes Theorem — the part of the equation that serves to update an existing, prior belief when forming some new, posterior belief.

Using mathematical symbols, the odds form of the equation (which is most relevant to this discussion), with the ‘components’ labelled, is as follows:
$$ \underbrace{\dfrac{p(H_{1}|E,\textit{I})}{p(H_{2}|E,\textit{I})}}_{\text{Posterior Odds}}= \underbrace{\dfrac{p(E|H_{1},\textit{I})}{p(E|H_{2},\textit{I})}}_{\text{Likelihood Ratio}} \cdot \underbrace{\dfrac{p(H_{1}|\textit{I})}{p(H_{2}|\textit{I} )}}_{\text{Prior Odds}} $$
where $H_{1}$ and $H_{2}$ represent two competing propositions (hypotheses, theories) of interest, $E$ represents the evidence observed and evaluated by the examiner, and $I$ represents relevant background (framework) information pertaining the matter. The concept of propositions is discussed more fully in another post that you can read here. It is important to note that the ‘flow’ in the decision-making process is from right to the left in the above equation, meaning that the prior belief about the propositions are modified by the likelihood ratio to form a new posterior belief based upon the evidence.

Now, before any mathematician or statistician has a fit while reading this, I am well aware that we are not generally working with true or real ‘odds’, and that it isn’t a true likelihood-ratio either.2 That stems, in part, from the fact that there is no requirement that the proposition set be exhaustive in nature. Competing propositions must be mutually exclusive — and ideally the set would be exhaustive but that isn’t necessary or even normal in a casework setting. The propositions that matter in a trial are those that represent the positions of the parties involved. There is no requirement for all possibilities to be considered by the trier — only those raised or argued by the parties. Nonetheless, while we are not talking about odds or a likelihood-ratio in the strict sense, the logic embodied in the above equation remains valid so long as all of the conditioning factors are clearly expressed.

The only part of the above equation that an examiner is actually able to evaluate, based on their expertise, knowledge and abilities, is the likelihood ratio.3 A key issue with application of Bayes Theorem, in its full form, is that examiners do not, as a general rule, have enough information to fully assess the prior odds.4

Without some accurate assessment of prior odds, it is impossible to apply the LR to determine or assess any posterior odds that may result.

It is the lack of priors that shows why the use of Bayes Theorem, in its full form, is impossible or impractical for our work, at least most of the time. I personally feel that it makes absolutely no sense to consider this option for regular casework though there may be times when an examiner has knowledge that could help to inform real ‘priors’ — for example, information about the prevalence of a particular type of printer in a given market.5 Furthermore, such an approach might be very applicable in some forms of research or other study.

The key to understanding this position is the realization that, in a research setting, a scientist is able to apply the full process to adjust their own (probabilistic) belief about the hypotheses being tested. In fact, that’s what they should be doing. This is completely different from real-world casework because such statements speak directly or indirectly to a matter this is properly left to the court. In that context, it is the trier-of-fact (judge or jury) who, in their role, must make decisions about the propositions and, ultimately, guilt vs. innocence (or liability). It is not the role of the expert to make any such decisions.

We must be very careful not to state conclusions in a way than might (not will, but might) influence the court improperly. Forensic science evidence can be very powerful and have a very strong influence on both judges and lay jurists.  Furthermore, we must remember that some judges/jurists would like to have someone ‘make the decision’ for them, even if that is not the way things should work — that attitude is simply human nature at play. Their desire to have such an answer should neither dictate the response nor cause the examiner to extend their opinion further than is warranted.

An approach based on evaluating, and reporting, solely the likelihood-ratio has been called various things, but I like to call it the ‘coherent logical approach’ to inference and evidence evaluation.6 The main reason I use this terminology is that the approach abides by all the rules of logic in a clear, coherent and understandable manner. Thus, the phrase is descriptive, accurate, and concise.


It also avoids the adjective ‘Bayesian’ which has come to hold negative connotations for some people — rather unfairly in my opinion, but so it is. At the very least, things ‘Bayesian’ are viewed with suspicion and considered by many forensic examiners to be controversial if not completely wrong. Indeed, many people (dare I say most?) have an opinion on the matter and, all too often, it is a rather negative opinion.

In my experience that kind of negative opinion comes mainly from a lack of understanding. But why fight about semantics? In light of the fact that the LR should be the focus, plus the fact the expert is NOT in a position to evaluate the whole theorem, why not just avoid the term? It is far better, in my opinion, to focus on the coherence and logic found in the approach (see also my post When is a Bayesian not a Bayesian?). Calling it the coherent logical approach also avoids the term ‘likelihood ratio’ which minimizes arguments with statisticians and mathematicians when you run into them in a bar.

Historically speaking, this approach to reasoning is nothing ‘new’. These concepts have a long history and were well-known in the 1800’s when ‘forensic science’ was only just developing as a recognizable discipline (and certainly wasn’t called forensic science by anyone). Today, there is what can only be called a ‘resurgence’ of interest and a push towards the use of a ‘Bayesian’ or ‘logically coherent’ approach for forensic evaluation.7 There are many groups involved in this effort with many of the key authors hailing from the University of Lausanne, but extending to groups like the NIST-sponsored OSAC initiative.

When discussing the evaluation of evidence one of the key concepts to grasp is that of uncertainty. All knowledge and information is, of course, limited. Although many people are completely confident that they know what has happened in the past, there is generally no absolute guarantee they are correct in that belief. The relationship between confidence and potential error is very important in forensic work.

This leads to the question of how uncertainty should be handled while being rigorously logical in our assessments. As it would be in any scientific pursuit this is achieved through the use of ‘probability’. Unfortunately, many forensic examiners become anxious as soon as either uncertainty and probability are mentioned. This is rather strange because examiners have always incorporated these concepts into their reasoning and conclusions though the standard itself speaks to confidence in our conclusions rather than potential for error or uncertainty. Forensic practitioners use probabilistic reasoning and wording in their conclusions, even if it is not explicitly stated or recognized as such.

If one accepts that a logically coherent approach is desirable or, at least, preferable in most situations, then the question becomes ‘how’ should an examiner be evaluating the likelihood ratio? There are three simple ‘rules’ that can help when applying logic properly for the evaluation evidence:

  1. There is ALWAYS a framework of information,
  2. We must always consider at least two competing propositions, and
  3. We must always evaluate the probability of the evidence (observations) given the propositions, and NOT the probability of the propositions.

I expand upon these three rules in another post; particularly the last one since there is a lot to discuss about how one assigns such probabilities and how any final opinion can/should be reached and explained. However, so long as an examiner keeps these points clearly in mind, they will be able to evaluate and explain the appropriate weight to be given the evidence.

Footnotes

  1. The term “likelihood ratio” is often used in this context but from a mathematician’s point-of-view a more appropriate term might be “Bayes Factor” since we are talking about something that has little or no quantified empirical data to support it. Our construct is driven and informed by the examiner’s belief based on their expert knowledge, training and experience. I am more concerned with the concept from the perspective of logical reasoning than as a practical tool driven by empirically-derived statistics. The latter will perhaps come in time and with research but, until we have such data, I advocate using this approach in the manner advocated by Lindley and in terms of subjective probabilities assigned by the examiner. Despite this distinction I will continue to use the term LR in this discussion.
  2. And, of course, at this point in time we are almost always working with subjective probabilities which which some statisticians seem to have trouble using.
  3. I should also mention that the likelihood ratio exists and functions independently of Bayes Theorem. That is to say, there is no need to invoke the Theorem at all in order to determine or use the likelihood-ratio, though it does provide a logically-sound mechanism for belief-updating based upon the likelihood-ratio. See DH Kaye’s article “Likelihoodism, Bayesianism, and a Pair of Shoes” (Jurimetrics, Vol. 53, No. 1, pp. 1-9, Fall 2012) for more information on this point.
  4. For fairly obvious reasons, a strong argument can be made that the examiner should not have such information, both because it may have a biasing effect on their evaluations and because it falls more properly in the domain of the court.
  5. It should be noted that information relevant to prior odds, such as base-rates, can be handled in more than one way. It might be presented to the trier separately to let them adjust their belief about the situation before they hear what else the evidence tells them. It could also be incorporated into the likelihood ratio itself. Or the examiner might use that information as a limited form of prior to generate posterior odds via the LR. Of these options, I favour presenting the information completely separately from the LR as it avoids the potential for ‘double-counting’ the evidence.
  6. The names ‘coherent logical approach’, ‘logical approach ’, ‘evaluative approach’, ‘evaluative reporting’ are all essentially synonymous. Even the ‘likelihood-ratio (or LR) approach can be used. Take your pick.
  7. And even that isn’t ‘new’ since it began over fifty years ago.

3 thoughts on “Introduction to the Logical Approach to Evidence Evaluation

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.