At Rocket Matter, we are constantly looking at metrics. Each functional area of our business collects, measures and evaluates relevant data.
We then take that data and try to turn it into actionable information – finding patterns, identifying opportunities, improving what we do.
Our metric focus notwithstanding, data alone rarely provides complete answers. Worse yet, looking at data without ascribing the proper context can yield imperfect – even misleading – conclusions.
Warren Buffet once gave perhaps the best lesson for this, in the context of evaluating investment skill. Once you see this simple example, it’ll stick with you whenever you see conclusions being drawn from large data sets.
I’m hoping that some of our trial attorneys might even get some mileage out of this example with a jury, or perhaps some of our appellate experts can draw a useful analogy or two to it in a brief.
Buffet’s Example: Imagine 225mm people decide to pair off and start a national coin-flipping contest, one flip per day. After each flip, the winner gets $1 and the loser drops out of the contest.
After 10 flips, about 220K people will have won a little over $1,000 – and will probably be feeling pretty lucky. Some may even start feeling confident. Undoubtedly, many will have started to explore what they’re doing to be so “good” at this task.
They may start flipping the coin in a certain manner, or craft some sort of pre-flip ritual to help them win, like always thinking “win” in their mind during the flip, or blinking exactly 5 times while the coin is in the air.
After 20 flips, we’ll have 215 people left, each with over a million dollars. At this point, there is no question that many of the 215 will deem themselves masters in the art of coin flip forecasting.
They may appear on CNBC, explaining their personal methodologies on successfully predicting coin flips. Some will be retained to publish popular books, like: “How I Made a Million Dollars in 20 days Without Working A Day.”
A few might even become celebrities, with people clamoring just to get a few minutes worth of personal, customized advice from these oracles. After all, they’ve correctly picked the result of 20 consecutive coin flips – there must be a secret, a replicable recipe for this amazing feat.
Of course there is, and it’s neither complex nor particularly intellectually satisfying. To those of us who love finding patterns in data, I’d even say it’s downright disappointing. The secret is: start with a big enough population.
(For an interesting read of Bayesian analysis actually literally applied in a criminal trial – which an English appellate court ultimately criticized – take a look at Regina v. Denis John Adams (1996).)
The point is simple, yet ridiculously important: data analysis is critical, but it’s rarely, if ever, the whole story. Phrases like “correlation is not necessarily causation”, and examples like those from Buffett and Brown are powerful reminders.
Steve Jobs himself hinted at this very idea, in a way that resonated far beyond the basic idea of being careful with data.
“You can’t connect the dots looking forward; you can only connect them looking backwards.”
Data is an incredibly useful tool, and collecting and analyzing it is an absolute must for just about process that is subject to potential improvement. But it’s just that – a tool – and must be balanced with context, experience, judgment, and vision.
So, before you ascribe too much credit to anyone, from the uncannily terrific portfolio manager to the leader in your office football pool, consider the context.