It is absolutely true that most forensic scientists want to be completely logical, open and transparent in their approach to the evaluation of evidence.  Further, I am sure that most document examiners believe this is exactly what they are achieving when they apply the procedures outlined in various traditional textbooks or the SWGDOC/ ASTM standards; for example, the SWGDOC Standard for Examination of Handwritten Items.

Given the very understandable desire to be logical, I find it strange that so many people have a negative attitude towards anything Bayesian in nature.  After all, an approach to evidence evaluation conforming to the Bayesian philosophy or approach would be quite literally the embodiment of logic (more specifically, probabilistic logic).

Expert witnesses serve an important role in any justice system.   The Criminal Code of Canada, for example, provides for such testimony in section 657.3 (1):

In any proceedings, the evidence of a person as an expert may be given by means of a report accompanied by the affidavit or solemn declaration of the person, setting out, in particular, the qualifications of the person as an expert if (a) the court recognizes that person as an expert and (b) the party intending to produce the report in evidence has, before the proceeding, given to the other party a copy of the affidavit or solemn declaration and the report and reasonable notice of the intention to produce it in evidence.

Unlike lay witnesses, expert witnesses are permitted and expected to express their opinion and give testimony about their belief regarding the matter at hand based upon special knowledge or ability that they possess.   In 1931, the Supreme Court of Canada, in Kelliher (Village of) v. Smith, [1931] SCR 672, quoted Beven on Negligence, 4th Ed., and wrote:

To justify the admission of expert testimony two elements must coexist:
(1) The subject-matter of the inquiry must be such that ordinary people are unlikely to form a correct judgment about it, if unassisted by persons with special knowledge.

(2) The witness offering expert evidence must have gained his special knowledge by a course of study or previous habit which secures his habitual familiarity with the matter in hand.

The topic was discussed further and expanded upon greatly in the seminal ruling R. v. Mohan, [1994] 2 S.C.R. 9

Ultimately, the first responsibility of any forensic expert (expert witness) is to evaluate evidence which falls within their scope of reference, in terms of particular and specific propositions of interest in a given case, while taking into account relevant background information.  Then the expert must convey the meaning of that evaluation, as it applies to the matter at hand, to any interested party.  Very often that will be initially an investigator or legal counsel. Sometimes it will proceed to a judge or jury in a court of law.  It is critical to remember the obvious point that, as a forensic expert our evidence is part, but only part, of the overall process whether it be a trial or simply an investigation.  In either situation, the decision to proceed one way or the other rests with that other party and generally involves a variety of evidence including our own.  That’s just another way of saying that all evidence should be considered by the trier-of-fact when reaching their final decision on disposition of the matter.  And, as experts, we rarely have access to anything beyond our little piece of the puzzle.  For obvious reasons, that is a good thing.

As such, it is my opinion that the focus of the expert’s work should be the determination and expression of a ‘likelihood ratio’ (or LR)1 that relates to the support provided by the evidence for each of at least two competing propositions, representing (as best they can) the positions of the parties in a legal dispute.  The LR is only one part of Bayes Theorem — the part of the equation that serves to update an existing, prior belief when forming some new, posterior belief.

Using mathematical symbols, the odds form of the equation (which is most relevant to this discussion), with the ‘components’ labelled, is as follows:
$$ \underbrace{\dfrac{p(H_{1}|E,\textit{I})}{p(H_{2}|E,\textit{I})}}_{\text{Posterior Odds}}= \underbrace{\dfrac{p(E|H_{1},\textit{I})}{p(E|H_{2},\textit{I})}}_{\text{Likelihood Ratio}} \cdot \underbrace{\dfrac{p(H_{1}|\textit{I})}{p(H_{2}|\textit{I} )}}_{\text{Prior Odds}} $$
where $H_{1}$ and $H_{2}$ represent two competing propositions (hypotheses, theories) of interest, $E$ represents the evidence observed and evaluated by the examiner, and $I$ represents relevant background (framework) information pertaining the matter.  The concept of propositions is discussed more fully in another post that you can read here.  The ‘flow’ in the decision-making process is from right to the left meaning that the prior belief about the propositions are modified by the likelihood ratio to form a new posterior belief based upon the evidence.

Now, before any mathematician or statistician has a fit while reading this, I am well aware that we are not generally working with true or real ‘odds’, and that it isn’t a true likelihood-ratio either.  That stems in part from the fact that there is no requirement that the proposition set be exhaustive in nature.  Competing propositions must be mutually exclusive — and ideally the set would be exhaustive but that isn’t necessary or even normal in a trial setting.  The propositions that matter in a trial are those that represent the positions of the parties involved.  There is no requirement for all possibilities to be considered by the trier — only those raised or argued by the parties.  Nonetheless, while we are not talking about odds or a likelihood-ratio in the strict sense, the logic embodied in the above equation remains valid so long as all of the conditioning factors are clearly expressed.

The only part of the equation that an examiner is able to evaluate (based on their expertise, knowledge and abilities) is the likelihood ratio.2  Examiners do not, as a general rule, have enough information to fully assess the prior odds.  And without some accurate assessment of prior odds, we cannot apply the LR to determine or assess any posterior odds that may result.

It is the lack of priors that shows the use of Bayes Theorem, in its full form, to be impossible or impractical for our work, at least most of the time.  I personally feel that it makes little sense to consider this option for regular casework though there may be times when an examiner has knowledge that could help to inform real ‘priors’ — for example, information about the prevalence of a particular type of printer in a given market.3  And such an approach might be very applicable in some forms of research or other study.

The key to understanding this position is the realization that, in a research setting, the scientist is able to apply the full process to adjust their (probabilistic) belief about the hypotheses being tested.  But, in real-world casework, such statements speak directly or indirectly to a matter this is properly left to the court. It is the court (judge or jury) who must, in their role as trier, must make decisions about the propositions and, ultimately, guilt vs. innocence (or liability).  It is not the role of the expert to make any such decisions and we must be very careful not to state conclusions in a way than might (not will, but might) influence the court improperly.  Forensic science evidence can be very powerful and have a very strong influence on both judges and jurists.  Furthermore, we must remember that some judges/jurists would like to have someone ‘make the decision’ for them, even if that is not the way things should work — that is just human nature at play.  Their desire should not dictate our response or cause us to extend our opinion further than is warranted.

An approach based on evaluating and reporting only the likelihood-ratio has been called various things but I like to say it is the ‘coherent logical approach’ to inference and evidence evaluation.  The main reason is the approach abides by all the rules of logic in a clear, coherent and understandable manner.  Thus, the phrase is descriptive, accurate and concise.

It also avoids the adjective ‘Bayesian’ which has come to hold negative connotations for some people — rather unfairly in my opinion, but so it is.  At the very least, things ‘Bayesian’ are viewed with suspicion and considered by many to be controversial if not completely wrong.  Indeed, many people (dare I say most?) have an opinion on the matter and often it is a rather negative opinion.  In my experience that kind of negative opinion comes mainly from a lack of understanding.  But why fight about semantics?  In light of the fact that the LR should be the focus, plus the fact the expert is NOT in a position to evaluate the whole theorem, why not just avoid the term?  It is better, in my opinion, to focus on the coherence and logic found in the approach (see also my post When is a Bayesian not a Bayesian?).  Calling it the coherent logical appraoch also avoids the term ‘likelihood ratio’ which minimizes arguments with statisticians and mathematicians when you run into them in a bar.

Historically speaking, this approach to reasoning is nothing‘new’. These concepts have a very long history and were very well-known in the 1800’s when ‘forensic science’ was only just developing as a recognizable discipline (and certainly wasn’t called forensic science by anyone).  Today, there is what can only be called a ‘resurgence’ of interest and a push towards the use of a ‘Bayesian’ or ‘logically coherent’ approach for forensic evaluation.  There are many groups involved in this effort with many of the key authors hailing from the University of Lausanne.

When discussing the evaluation of evidence one of the key concepts to grasp is that of uncertainty.  All knowledge and information is, of course, limited.  Although many people are completely confident that they know what has happened, there is generally no absolute guarantee they are correct in that belief.  The relationship between confidence and potential error is very important in our work.

This leads to the question of how we should deal with uncertainty while being rigorously logical in our assessments.  As it would be in any scientific pursuit this should be done using ‘probability’.  Unfortunately, many forensic examiners become anxious as soon as thing like uncertainty and probability are mentioned.  I find this rather strange because examiners have always incorporated these concepts into our reasoning and conclusions (though we speak to confidence in our conclusions rather than potential for error or uncertainty).  Forensic practitioners use probabilistic reasoning and wording in their conclusions, even if it is not explicitly stated or recognized as such.

If one accepts that a logically coherent approach is desirable or, at least, preferable in most situations, then the question becomes ‘how’ should an examiner be evaluating the likelihood ratio?  There are three simple ‘rules’ that can help when applying logic properly for the evaluation evidence:

  1. There is ALWAYS a framework of information
  2. We must always consider at least two competing propositions, and
  3. We must always evaluate the probability of the evidence (observations) given the propositions, and NOT the probability of the propositions

I expand upon these three rules in another post but, if an examiner keeps them clearly in mind, they will be able to evaluate and explain the appropriate weight to be given the evidence.


Footnotes:

  1. The term “likelihood ratio” is often used in this context but from a mathematician’s point-of-view a more appropriate term might be “Bayes Factor” since we are talking about something that has little or no quantified empirical data to support it.  Our construct is driven and informed by the examiner’s belief based on their expert knowledge, training and experience. I am more concerned with the concept from the perspective of logical reasoning than as a practical tool driven by empirically-derived statistics. The latter will come in time with research but, until we have the data, I advocate using this approach strictly to guide the evaluation process. Despite this distinction I will continue to use the term LR in this discussion.
  2. I should also mention that the likelihood ratio is an independent concept and we do not have to invoke Bayes Theorem at all in order to use it.  See DH Kaye’s article “Likelihoodism, Bayesianism, and a Pair of Shoes” (Jurimetrics, Vol. 53, No. 1, pp. 1-9, Fall 2012) for more information on this point.
  3. It should be noted that information relevant to prior odds, such as base-rates, can be handled in more than one way. It might be presented to the trier separately to let them adjust their belief about the situation before they hear what else the evidence tells them. It could also be incorporated into the likelihood ratio itself. Or the examiner might use that information as a limited form of prior to generate posterior odds via the LR.  Of these options, I favour presenting the information completely separately from the LR as it avoids the potential for ‘double-counting’ the evidence.

Leave a Reply

Your email address will not be published. Required fields are marked *