Statistics — Worse than a lie

It has oft been said that “there are three kinds of lies: lies, damned lies and statistics”.  That phrase, according to Mark Twain, came from Benjamin Disraeli. Interestingly, it has never been found in Disraeli’s written works so that attribution is likely incorrect.

A lie, perhaps, by Twain?

Statistics lies, Mallett 2006

But I digress.  The source of the statement doesn’t really matter.  It is enough that the phrase reflects the belief that many people have when they think about statistics.1 It is a catchy little phrase.  Yet most reasonable people know that numbers — and statistics are simply numbers after all — cannot do anything on their own.  Hence, statistics can no more lie than they can sing or dance.

Statistics are a tool.  Every tool has a purpose for which it works well.  When used correctly and properly they are generally safe, effective, and beneficial.  But, if misused, tools can also cause a great deal of damage.  It is important to understand what statistics are, what they mean and how they can be used, or mis-used.

Let’s start from the beginning.  Mathematicians (and even moreso, statisticians), being a truly lazy lot, don’t like to write out long lists of numbers, especially when there is no need.2  They certainly don’t want to spend hours upon hours working at a desk to make sense of the numbers when there is a perfectly good pub waiting for them just down the street.  No, it is easier (and sufficiently accurate for many things) to summarize sets of numbers using a few well-chosen descriptors. Those descriptors are statistics and, in their most basic form and use, they serve as a type of numerical short-hand.  When used correctly, they are a convenient way to describe a (large) set of numbers using far fewer numbers.  There is nothing wrong with that at all.

Of course, even in this generally-benign application it is quite possible to describe a set of numbers incorrectly or inadequately if you aren’t careful.  Suppose you want to know the ‘average’ value for a set of numbers.  That’s a handy thing to know sometimes.  The problem is that there are lots of different ‘types’ of average values that might be calculated for any given set of numbers.  The best choice really depends upon the nature of the data and what you need to know about the data.  Without going into details you might considering looking at the mean, the median or the mode — all common and legitimate measures of ‘central tendency’ (among many others).  Each describes a different aspect of the data set and any one may be the best choice depending upon your needs.  In order to pick the right tool to use, the tool-user really should know something about their tool, don’t you think?3  Is it right or fair to blame the tool, or should we blame the person using it?  Should we really blame the mis-representation that results from that person’s inability to choose or use statistics properly on the numbers or statistics?

Beyond description, statistics can also help if we want to prognosticate, to test theories or beliefs about a concept or idea.  This invokes the concept of statistic inference which is the process of drawing conclusions from data that varies.  To use stats in this way, the concept or idea must first be quantified (because we need numbers).  Many statistical tests or comparisons can be performed, once we have quantified data, all of which depend upon certain assumptions being true.  The so-called ‘parametric’ tests tend to be more ‘powerful’ but the data must fulfill many specific assumptions relating to the underlying statistical model being applied before the test can be considered valid.  ‘Non-parametric’ tests make fewer assumptions (little or nothing is assumed about the structure of the data or underlying population) but, as a result, these tests are not as powerful in terms of detecting a ‘real effect’.

And, ultimately, the outcome of any test (parametric or non-parametric) depends entirely upon the nature of the numbers themselves; that is, what do the numbers really mean?

Consider the following two sets of numbers:
Set A:  80.98, 80.14, 80.12, 80.24, 80.33, 80.83, 80.94, 80.67
Set B:  79.92, 79.94, 79.98, 79.97, 79.97, 79.73, 79.95, 79.97

Now, let’s ask the question are these sets the ‘same’?  Are they ‘different’?  Well, they are obviously not the same, right?  On a quick glance (and even after a bit of reflection) it is hard to describe exactly how the two sets differ even though the numbers look different. So let’s simplify things a bit.  We might, for example, look at the mean of each set.  The arithmetic mean (and, yes, there are lots of different types of means to pick from, too) of A is approximately 80.53 and the mean of B is approximately 79.93.  So now we can say that the mean values are different.  But maybe we want to know if the numbers are samples taken from some common, larger group or population.  In that case, we might test further by doing a simple t-test on the two sets.  We are still looking at the means but now we have to also considering the spread or variance in the sets.4

Using R, a handy open-source software package, we see from a t-test that these two sets are statistically different at a level of 0.001.  That’s cool.  We now have a stat telling us two other stats are statistically different.5

But so what?  What are these numbers?  And what do these “statistics” mean? Does this difference actually matter?  If the numbers represent (for example) the temperature of two rooms in our office, they would mean absolutely nothing in any practical sense.6  If, however, they are the temperatures in two thermally-controlled environmental rooms, then the difference might actually be important.

In reality, they are just a bunch of numbers I made up and they mean absolutely nothing. Think about that for a minute…

Yes, statistics can be generated using any numbers — statistics don’t care any more than they lie, sing or dance.

Over the years I have heard people say that statistics are meaningless unless they are used to present, defend or refute an argument, position or belief.  To a degree that is a valid point-of-view but I also think it is part of the reason “why” people believe statistics lie.  People do not, as a general rule, like to think that they are either gullible or unable to reason logically.  Yet those same people tend to accept statistics at ‘face value’ without challenging them, or at least challenging the basis for them.  As a result, most people encounter statistics in arguments and consider them to be the backbone or essence of the argument.  Then, any time an argument is shown to be wrong or false they cannot accept that they themselves might be at fault for accepting the argument (and the stats) in the first place.  Oh no, it must be the statistics that lied.  Indeed, it wasn’t just a lie, it was a damned lie.  No, wait, it was statistics.

Interestingly enough, the person who presented the statistics in the first place is often overlooked in the whole discussion…

Ultimately, numbers and statistics do not lie and cannot lie; people do.  Anyone who uses a statistic to ‘prove’ a point should be prepared to present proof that the numbers are valid and reliable, in the context of interest.  Indeed, one might argue that this proof should be provided together with the numbers (the actual data) and statistics, up-front and without request from the recipient of the information.  But, while I wholeheartedly agree with that suggestion, I would add that anyone misled by statistics need look no further than a mirror to see the real problem.  People rarely consider what a statistic means or whether a provided statistic was appropriate and valid.

Now that places the error on the recipient of the information.  I have heard it suggested that it isn’t fair to expect people to ‘vet’ or challenge information they have been given.  I would counter with the suggestion this is the price to be paid when you live in an ‘information age’.  We are inundated with information, data, statistics of all type and manner.  Why in the world would someone think that everything they hear or read is ‘the truth’ or unbiased?

It’s very clear that some people use statistics with malicious intent knowing full well that their audience is gullible and uninformed.  Those people are clearly at fault for doing so and they are little more than fraud artists.  Others may do so through ignorance — they don’t even know what they are doing is ‘wrong’. Often it’s an example of a little knowledge being a dangerous thing.  Those people can perhaps be given the benefit of the doubt but I don’t think they should be considered completely blameless.

In either case, the statistic is not ‘at fault’.  The fault lies with the person presenting the statistic as part of their argument or description.  And some of the fault lies with the recipient of the information who failed to challenge the statistic or the argument properly.

If the reader is honest with themself (and not too put off by my comments), I suspect they might agree with what I’ve said here.  But I also suspect that most people will continue to condemn statistics — because, like mathematicians, they are fundamentally lazy.  Too lazy to learn enough about statistics to defend themselves.

When next you hear that phrase, consider my variant on it.  It goes like this; “there are lies, damned lies and those who believe the lie when they should know enough to question it in the first place.”

Try to avoid being one of the latter… and please don’t blame the statistics.


Footnotes:

  1. Of course, anyone interested in such things can always find more on the web. Check out the History of Statistics page at the University of York. Yes, there is an entire page devoted to the famous statement which they attribute (sort of) to Sir Charles Wentworth Dilke.
  2. Lest you think I’m off-base and being cruel with such assertions about the laziness of math and stats gurus, I will provide two pieces of supporting evidence.

    First, I think I fall in this category… well, sort of. I started out my university ‘career’ studying physics and math (double honours no less). That lasted 3 years which probably means that I am less lazy than the rest of them. But in that time I learned from direct and personal experience that most math, and pretty much all stats, are intended to simplify and organize information. That is, in a very real sense, the purpose of statistics — take a large volume of information and make it easier to understand. And make it disclose things you would not see otherwise. But I personally think that function is a mere by-product of the essential laziness inherent to those doing the work in the first place.

    Second, and far more pertinent, my eventual thesis advisor was one of the few people I ever met with a PhD in mathematical psychology. Yes, there was such a thing though it has changed over time to various other names. Professor Bill Petrusic was a brilliant, fascinating and very insightful person and I distinctly recall him pointing out to students in his stats class that “statisticians are lazy guys”.

    And, yes, the word “guys” in this context is accurate.  Remember that, at the time, there were very few ladies in the field so this wasn’t a particularly sexist remark.  In fairness, there were a few female students in my classes but I think it is safe to say that those ladies who were studying statistics were different. In particular they were not lazy like the guys. Which probably explains why they were much better at it than the rest of us.

    So, yes, I really was taught this at university (or, quite possibly, at the pub after class). As a formal reference, best to consider it a personal communication, c. 1979).

  3. A rather cool resource for those interested in such things is shodor.org which provides online information appropriate for all levels of understanding from grades from 3 to 12 and even some good university level material. The ‘Interactivate’ section of the website is particularly interesting; check out the ‘Discussions’ section for Instructors. It probably isn’t the best all-around statistics reference site out there but it has a lot of info. For example, check out this interactive discussion of the Mean, Median, and Mode.
  4. And making a lot of other assumptions about the data which we will happily ignore.
  5. For a nice discussion of significance and p-values, please see Jim Frost’s blog post “Understanding Hypothesis Tests: Significance Levels (Alpha) and P values in Statistics”
  6. So long as we are talking in degrees Fahrenheit. If they are in Celsius likely there is a problem.

Leave a Reply

Your email address will not be published. Required fields are marked *