Lies, Damn Lies and Statistics

Every day through the paper, TV, magazines and internet we are inundated with the latest statistics on just about everything. From the impact of computer time to childhood obesity numbers are thrown at us at an overwhelming pace. How do we sort through it all? With a few simple principles and questions we can make a start of it at least. Let's give it a go by trying to dissect this statement:

"Local farmer experiences a 50% reduction
in his cattle herd."

1. Know your Baseline

• What if I told you this farmer only had 2 cows in his herd? Doesn't sound so horrific anymore does it? Without knowing where you are starting from it is hard to make a judgment about where you are now.

2. Statistical significance vs Practical Significance

• A 50% reduction seems significant. It sounds like a really big number however without context it is hard to determine whether it is profound beyond the numbers. A reduction from 2 cows to one cow hardly seems noteworthy. Which leads us to ...

3. "The So What?" Factor

• This is the question I ask to figure out whether we are even measuring the right thing. I'm sure we all think that cows are lovely -- a summer BBQ just wouldn't be the same without them -- but I think we are more concerned with the farmer's livelihood. The reason that statement sounds horrific is because we associated it with the farmer losing half of his income or assets. That may or may not be true but we need to look at....

4. Who is Being Studied

• What if this farmer was in a Third world country and his entire wealth was two cows? What if the cow that died was the sole source of protein (via milk) for his children? or What if this was a hobby farmer who had cows just to keep the grass down on his half million dollar acreage? Conclusions that we arrive upon have to take into account who the statement originally referred to. We can't always extrapolate from one population to another.

This is by no means and exhaustive list of questions to ask when you are evaluating the validity of a statement drawn from a study but it's a place to start. I hope the next time you hear about a statistic that claims increased test scores due to a certain intervention (or any other statistic for that matter ) you will remember the farmer with a herd of 2 cows and ask yourself questions like:

• What were the test scores to begin with?
• Is that an increase that sounds significant only in numbers or in real life too?
• Is test scores what we should be measuring? Should we be measuring something else like success after School?
• Who did they measure this increase in? Is my child in that same environment? Are those results transferable?

With careful questioning, the statistics can then tell a story that is no longer a lie or a damn lie but something that is closer to the truth.