Startup entrepreneurs solve problems and get paid for their solutions. But when analyzing a problem, ensure you have the right solution based on the data you collect.
During WWII, Allied bomber losses were high, so high that the British Air Ministry undertook a rigorous analysis in hopes of finding a solution. Their engineers set out to examine every bomber they could, gathering data on each bullet hole. After analyzing the results, engineers decided to reinforce the areas that had the highest concentrations of holes with armor plating.
It didn’t work.
Perplexed, the engineers assumed that the extra plating had made the planes too heavy, and that the difficulty in handling the planes was offsetting the protection of the armor plating.
Enter Abraham Wald.
Wald, a mathematician, suggested that they simply put extra armor plating where the bullet holes weren’t. The idea was simple: if the planes are returning with bullet holes, obviously those areas can be struck without causing the planes to crash. The planes that weren’t returning, Wald theorized, are the ones that are getting hit in different areas.
The engineers’ error was so significant, statisticians decided to name it: Survivorship Bias (the tendency to include only successes in statistical analysis). Or more appropriate, use Statistical Process Control (from W. Edwards Deming) in order to understand statistical variation. Pay particular attention to his funnel experiment using statistics.
Any time you only examine just the successes, you will skew the results.
So what is our take away from this story?
- Don’t assume something is happening, collect data to help identify where the problem lies.
- Don’t collect partial data (as seen from the above example), collect all of the data about the problem. Parents and managers do this all of the time, collect one piece of information and change everything based on one example.
- Solve the problem, then follow the QA (Quality Assurance) cycle: Plan, Do, Check, Act.
- Plan your actions
- Do your actions
- Check your actions to see if it solves the problem
- Act on what needs to change, i.e. it may not solve the problem, then go back to #1 and redo the process.