Everyday Problem Solvers
As the 1940’s air war in Europe intensified, the Allies faced a major problem. Their bombers would leave England by the hundreds, but too many of them didn’t return, brought down by extremely heavy enemy flak. The Allies desperately needed to beef up the armor on their planes to provide protection, but armoring an entire plane, or even an entire cockpit, involved far too much weight. How could they choose the few especially vulnerable places to be armored?
A couple of clever engineers solved this problem with a counter-intuitive analysis. After comprehensively logging the locations of flak damage inflicted around the fuselages, engines, and cockpits of planes returning from hundreds of bombing runs, they calculated which particular locations had sustained an unusually low number of hits, and began armoring those areas for future missions.
In retrospect, their reasoning should be obvious: Flak bursts explode randomly all around, so the only reason returning planes have particular areas with less damage must be that the planes sustaining damage in those areas were less likely to return.
This story holds an important lesson for us when it comes to learning from the business successes and failures of the past, whether we’re talking about case studies and best practices or the best-selling books written by big-name business gurus. We cannot isolate the secret to any business’s success unless we first know why others just like it were not successful.
But nearly every classic best-selling business book, from Tom Peters’ In Search of Excellence to Jim Collins’ Good to Great, relies on success stories to draw out important lessons for others. Even though the authors clearly have the best of intentions, such books suffer from what academics call an “undersampling of failure.” While anecdotal evidence makes for more interesting reading than statistics, choosing the right anecdotes does matter.
To understand the perils of drawing lessons only from studying successes, suppose there is a particular business strategy that is highly risky, and across all companies that try it, 10% are very successful, but 90% fail outright. Now what if the successes are documented and studied later, while the failures are not publicized and simply drop from sight? Do you see the problem with this?
To be able to say with any confidence that a business is likely to be successful with a strategy based on past experience, you have to know not just how many succeeded by following this strategy in the past, but how many followed the strategy and failed. If the Allies had only patched the places where they saw the most flak damage on their returning planes, they would have done little to improve their chances.