Journal: News & Comment

Bad brains

Permalinks to this entry: individual page or in monthly context. For more material from my journal, visit my home page or the archive.

Probability and statistics are powerful arts, but the human brain seems poorly designed to deal with them. It's not hard to find examples: confidently buying lottery tickets, or driving an SUV because it's "safer," while being reluctant to fly on a major jet airline, or afraid of a stranger taking your kids at the playground, for instance.

Jakob Nielsen gives a good analysis of why statistics can be deceptive in web usability studies, but his explanations apply in many other fields as well. Particularly notable:

Researchers often perform statistical analysis to determine whether numeric results are "statistically significant." By convention, they deem an outcome significant if there is less than 5% probability that it could have occurred randomly rather than signifying a true phenomenon.

This sounds reasonable, but it implies that one out of twenty "significant" results might be random if researchers rely purely on quantitative methods.

and:

Quantitative studies must be done exactly right in every detail or the numbers will be deceptive. There are so many pitfalls that you're likely to land in one of them and get into trouble.

If you rely on numbers without insights, you don't have backup when things go wrong. You'll stumble down the wrong path, because that's where the numbers will lead.

 |