Applying the logical approach to evidence evaluation in forensic work can be a challenge. I am the first person to admit that is the case. However, I would argue that (1) a challenge is not something to be avoided — it’s something to take on, and (2) the traditional way of doing things is not easy — it too is a challenge. In fact, it is very important to understand that the logical approach is no more difficult or challenging than what has always been done by forensic document examiners. Indeed, in many ways it is simpler and more straight-forward than our ‘traditional’ approach to such things. This post will touch upon some of the practical issues faced by examiners when applying the logical approach to FDE work.
After studying and using this approach for some time I can assure you that it is not a particularly difficult approach to apply. In select aspects it differs from the traditional approach and, as a result, it feels unfamiliar. Often, this relates to the need to consider things more fully; giving consideration to things we tend to ‘quickly assess’ (or dismiss) when using the traditional approach. But it does not change most things that we do. In fact, it primarily affects the evaluation element of the work; not the elements of analysis or comparison. It requires a different way of thinking about the evaluation process which starts with the focus of the evaluation.
The earliest workshops I presented on this topic were focused, for the most part, on the theory and philosophical bases of the approach, plus the benefits of using it. I did not try to delve into the practical side of things, mainly due to time constraints, but also because I was simply trying to convey its value. FDE’s are, however, a practical lot so I feel that my approach has had limited impact. The most common feedback has been, “I can’t see how it will work in practice…”
That is a fair enough criticism, all things considered. More recently, I have been involved in workshops designed expressly for hands-on practice — after all, there really is nothing like hands-on activities to make something real and tangible. More of those workshops are needed, but they are time-consuming to create. For now, here are some thoughts on these issues deriving mainly from feedback received to date during those workshops.
I have adopted a basic Q&A format in this post because most of these topics were raised as questions during workshops, or as posts in public fora.
However, before I delve into specific questions and issues, if you are not already familiar with the logical approach to evidence evaluation you may wish to read at least one of my other posts. I have written a brief “Introduction to the Logical Approach to Evidence Evaluation” you can find here.
Now, on to the questions which include common objections and issues with this approach as voiced by qualified FDEs expressing their views in a public forum:
I seriously question this assertion. Does our traditional approach actually “work”, or is it more a matter of it being, at least most of the time, accepted in court?
In my experience, many examiners feel that a method or approach is fine and ‘works’ so long as it is accepted in court. I could accept that as a reasonable criteria if the courts properly vetted and controlled forensic evidence. But their track record in that regard is poor. Understandably so, I would argue. Is it realistic to expect a non-scientist/non-expert to judge the claims of an expert presented to them by one side or the other? Not really, but that’s what happens every time an expert is presented to the court.
The issue is that many people, certainly most examiners and many courts, do not even realize there are flaws and issues hidden in our present approach. They are, in essence, ignorant of the limitations in the methods and/or the fact that their testimony may well be misunderstood or mis-applied in court. Lacking that awareness they understandably find this approach perfectly acceptable. I think most people would agree it is critical to use methods and terminology that 1) provide accurate and reliable results, 2) are accepted in court, and 3) are properly understood by the court. Our traditional approach is routinely challenged on the first and third of these. As a rule, we have not had too many issues with the second.
I don’t know of anyone who is suggesting that every and all courts want us to change what we’re doing now. In fact, very few have expressed concern or recommended change of any sort. Yes, there have been a few such comments, but they are often hidden in a particular ruling or appeal. Most of the time the issue relates to overstating of evidential value or, for some DNA cases, transposition of the conditional (i.e. the prosecutor’s fallacy). So I would not characterize this as being pushed, driven or promoted by the courts. That may change as more courts become aware of the issues.
But, for now, the impetus for change comes from from academics, researchers and, yes, some practitioners. In addition, there have been recommendations from a variety of professional bodies that address these issues. In general, those recommendations all endorse the use of the logical approach to evidence evaluation. Those “pushing” this approach recognize its benefits—logical coherence, ease of transparency, focus on evidence, and avoidance of transposition.
I think of this as an attempt, likely unintentional, to mislead the trier. That may sound critical or extreme; however, when an examiner says this sort of thing they are suggesting two key things. First, their opinion is somehow special just because it is their personal belief, and, second, the apparent strength of the opinion (i.e., a definite conclusion) should be understood as something less than definitive because it is ‘just’ an opinion (i.e., read my mind, please). And the latter seems to be done with the ultimate objective of having the trier give more value to the opinion than is warranted, rather than conveying the actual weight directly and clearly.
Let’s dissect this further. First of all, to say that it is “just an opinion” is irrelevant. Anything and everything an expert witness says or expresses is “an opinion”. That is all it can ever be and it’s the reason why expert evidence is called “opinion evidence”. The issue here is the basis for the opinion which needs to be explored and clarified in terms of the case at hand. An expert opinion is based upon the knowledge, skills, and abilities of the examiner. It is not some arbitrary thing, but based upon the individual’s actual expertise.
Second, saying that the opinion “shouldn’t be taken as 100% certain” might seem like a reasonable thing but, at the same time, there is a clear implication that it ‘should’ be taken as pretty darn close to 100%. This assertion is made without any accompanying justification (because there is none), or explanation. It seems that, in general, examiners ‘expect’ others to understand this qualification of the opinion without any further direction or explanation; people should just understand. This type of thing is an attempt to side-step the real issues in order to continue expressing ‘definite’ conclusions even though such opinions are not warranted (as some courts have already ruled, and most experts would agree).
A key issue with any ‘definite’ opinion rests in its potential to usurp the authority of the court through excessive influence on the trier’s valuation. In addition, there is the issue that the examiner does not have (and generally does not want to have) the relevant information needed to make such a declaration. This type of answer ignores those issues completely.
First, strictly speaking, a likelihood-ratio is a numeric construct that does, in fact, require numeric data. However, there are different ways to produce meaningful numeric values to form a likelihood-ratio.
Quantified data can be obtained through experimentation and there are many ways to do this. Some are easier than others, but each has its own challenges relating to the sampling being used and other factors.
However, perfectly legitimate and valid numeric data can be based on personal, subjective and, most important, “expert” belief. To do this, a measure of probabilistic belief must be elicited from the examiner. Again, there are various ways to do this, including simply having an examiner express the level or strength of their belief. There are, of course, issues doing this (not least the fact that the brains of some examiners might explode from the effort), but it can be done. Some examiners do this now (…assign probabilities, not explode).
The resulting probabilities can be used just like probabilities informed through empirically-derived frequencies to form a type of likelihood-ratio. The basis of such a result is different, but no less valid.
Second, most people who had heard me speak on this topic will know that I try not to invoke the term ‘likelihood-ratio’ too often. I use it mainly for convenience because it is the term others like to use. I try to focus on the concept underlying the likelihood-ratio which relates to the value of information when updating one’s belief about something. In other words, I am concerned with the ‘concept’ of the likelihood-ratio, more so than the formal definition of it. I will sometimes mention the related concept of a “Bayes Factor” which some people prefer to use.
In general, I consider Bayes Theorem and its component elements to be a useful model as a system for reasoning that helps us to clarify issues that exist when doing this in a forensic/court context. A key point is that model works whether or not we have numeric data.
Now, before anyone starts screaming I should point out that this is very much in line with the traditional approach to FDE work. After all, the terms outlined in the ASTM/SWGDOC guideline for opinions have always been probabilistic in nature, yet they are defined in utterly non-quantitative terms. Is that an ideal way to do this? Or proper? Or correct? Perhaps not, but it’s certainly possible to do this sort of thing.1 It is the way we’ve always done this sort of thing, albeit while also taking things a bit too far by expressing transposed or definite/conclusive opinions. The important point is that this shows we are able to do things in this manner.
All we need to do is focus on the evidence, rather than the proposition, to properly conform to the logical approach. The beauty of the logical approach is that it compels the examiner to express belief about the evidence in a logically sound and coherent manner relating the evidence to the relevant propositions, and in the context of a case. In addition, when numeric data does become available to us it can be incorporated, and used, easily and effectively in the same construct. The same cannot be said of our traditional approach.
Well, it’s a good thing, then, that we are not using or explaining Bayes Theorem, isn’t it? This sort of question is mid-directed because forensic examiners do not and should not use or apply Bayes Theorem, in full, at all. The examiner should focus on evaluation of the evidence given the propositions of interest; in other words, expression of the likelihood-ratio or its equivalent.
More generally, I have no doubt that jurors vary in their ability to understand technical or expert testimony. That is one of the reasons why our job can be difficult and challenging at times. I have never had any particular issue or difficulty explaining these concepts to anyone. I seriously doubt that the people in my part of the world, or jurors in “my courts”, are any more capable than those from some other jurisdiction.
Really?
Sure, I agree that this is going to require some explanation, but anyone who thinks our present approach and conclusions are simple or easy to understand without explanation probably hasn’t given much testimony. Every time I have used our ‘traditional’ conclusions I have had to define and explain what the wording means. The situation is exactly the same when using the LR construct and, quite honestly, it is a lot harder to explain our traditional conclusions than it is to explain the LR construct, or the logical approach.
After all, how hard is it to understand a statement like, “the evidence more strongly supports the proposition that the suspect wrote the questioned signature, rather than the proposition that someone else wrote it”?2 Or, “the probability of seeing this evidence if the suspect wrote the questioned signature is very high, but extremely low if the signature had been written by anyone else”?3
Neither of these is a particularly complicated statement, nor hard to understand. Yet both of those are expressions that conform, literally, to a ‘likelihood-ratio’ construct. There are other choices of wording that are even simpler which, in a given case, might be used.
Now, there are also lots of people are working out the ‘best’ wording to be used. However, the key point is that none of these are difficult to understand.
I suppose this might be true…. if the examiner is utterly incapable of communicating relevant concepts and ideas.
In my experience, most examiners are bright people and excellent communicators. Effective communication, both verbal and written, is a key-stone of our work. I have absolutely no doubt that an examiner can learn to use this approach, apply it in their work, and explain it perfectly well.
In fact, in many ways this approach is easier and simpler than what we’ve always done. It really is. At the same time, it is a bit ‘different’ which may pose a challenge for some people, but it’s not an insurmountable challenge. Not at all.
In my opinion, this assertion just doesn’t apply to most examiners. Frankly, it seems very unlikely to me that any examiner would have a difficult time being understood.
The terminology can be confusing, but that’s only because it is unfamiliar.
Education is always a challenge, but not an insurmountable challenge.
I don’t expect anyone to use this approach—by which I mean apply it to all facets of their work, including testimony—without training. Anyone who advocates for the use of the logical approach knows that education is important for both examiners and their clients.
This type of criticism is based on a couple of mistaken beliefs.
First, that our traditional approach is perfectly fine in this regard. That belief leads to the idea that moving to the logical approach would be a step backwards, in terms of validation. The thing is our traditional approach is not extremely well-validated. Some validation studies have been done and they support the discipline nicely. Validation, however, is an ongoing process and it isn’t a matter or being, or not being, validated.
Second, it is important to remember that the logical approach uses the same knowledge base and analytical method as our traditional approach. The key thing that changes is the logic and reasoning in the evaluative component of the process. Therefore, this isn’t a new method at all, just another way of evaluating the findings… and expressing the final opinion. It should also be noted that, with a few exceptions, validation studies done to date have not been tied to any specific conclusion or opinion scale.
What this ultimately means is that whatever validation has been done for our present methods applies reasonably to the logical approach, with a few basic assumptions.
Invoking a lack of validation as a reason to not use the logical approach makes no sense because (1) we never worried about it for the traditional approach until recently (and still haven’t done much about it formally for the various methods we use), and (2) most existing validation studies apply just as well to the logical approach.
I will note that the opinion scale used for this type of opinion cannot, in and of itself, be invalid so long as it is properly formed scale with a clear basis and definition. The scale covers the full range of possible outcomes with each level clearly differentiated from all others. As a construct to be used by an examiner to express their belief about the matter at hand, there is nothing about the scale itself that can be considered invalid.
At the same time, there remains the issue of how well an examiner (or different examiners) can use such a scale. That particular issue exists for our traditional conclusion scales as well (and those scales suffer from the additional issue of being logically problematic). Finally, the logical approach produces results that are easier to validate than our present approach.
This is an interesting suggestion that requires its own blog post… coming soon.
Terminology is important and, in this case, it matters… a lot.
These three terms are not inter-changeable and, of them, ‘explanation’ is the most problematic to use for this purpose. For more information, see this blog post, and particularly this section.