On March 10, Ethiopian Airlines Flight 302 crashed outside of Addis Ababa, killing all 157 passengers and crew. This followed October’s crash of Lion Air Flight 610 in Indonesia, which killed all 189 onboard. Both crashes involved Boeing’s 737 Max 8 jetliner.
China, Indonesia, and Ethiopia grounded the aircraft on March 10, suspecting the crashes were caused by technical problems. The European Union followed on March 12. The Federal Aviation Administration grounded all U.S. flights the next day.
Investigators revealed that the crashes likely were caused by mistakes in the plane’s software, which pushed the planes into uncontrollable dives because of bad data from a single sensor. Evidently, no software or hardware redundancies were in place.
Bloomberg reports that the Max Maneuvering Characteristics Augmentation System (MCAS) and flight-testing software, was in no small part written and developed by Indian subcontractors who had little or no prior experience in the aviation industry. Why would Boeing hire Indian subcontractors to tackle critical tasks? Because American engineers cost between $40 and $80 an hour. Their Indian replacements, in contrast, worked for $9 an hour.
Boeing assumed that by moving its production to the Third World, it could exploit cheap foreign labor. This would allow Boeing to undercut its competitors, thereby gaining market share, boosting profits, and benefiting consumers all at once. No downside.
But that’s not how the world works. Everything has a price. In this case, while Boeing saved money on labor it also burdened itself with extra complexity and exposed itself to more “O-ring” vulnerabilities.
My Kingdom for an O-Ring
In 1993, Michael Kremer, a Harvard-educated developmental economist, wrote a paper called “The O-Ring Theory of Economic Development” in which he tried to explain why workers in some countries earn exponentially more than workers in other countries, despite doing the same job.
The theory also sheds light on how complexity can destroy not just a business, but the economy itself. Let’s start with an example.
Pretend you own a factory that makes glass vases. Two workers can make one vase: one blows the molten glass while the other paints the vase. If the vase is dropped then it shatters and becomes worthless.
You hire four workers. Two never drop vases and two drop them half the time. How should you divide your workers for maximum productivity? We instinctively want to pair a good with a bad worker—each team will have someone competent guiding it. Bad idea. If you do this then both teams will break half the vases.
Instead, you should pair the good workers together and let the droppers make a mess. Why? Together the good workers would succeed 100 percent of the time, while the droppers would succeed 25 percent of the time. On average, the teams would make vases 62.5 percent of the time—much better than half!
There are two lessons worth noting. First, vase production is fragile—a mistake at any point in the production-chain destroys the whole vase. This means that the chain is only as strong as its weakest link. Second, increasing the production chain’s complexity will increase the fragility in a nonlinear way. How?
Imagine you must also paint the vases. Although the two good workers succeed 100 percent of the time, they’re now forced to work with a dropper. Suddenly the factory’s productivity decreases to just half—one dropper ruins everything.
Now pretend you must varnish the vases too, adding a fourth step. Because your last worker is also a dropper, the factory’s output plummets to 25 percent.
Finally, let’s pretend that it takes 100 steps to make a vase. You hire 97 employees who always succeed, but three droppers slip through the cracks. In this case, your factory’s efficiency would decrease to just 12.5 percent—despite the fact that you have 97 perfect workers!
Thankfully, much of the economy is not subject to O-ring vulnerabilities because precision isn’t always critical, and many times mistakes can simply be fixed. For example, if a baker adds too much salt to his dough, the bread may taste salty but it will still be edible.
This is in stark contrast to many technologically-sophisticated products—like commercial jets—whose value can be erased by a mistake anywhere along the production chain. In Boeing’s case, a software failure relating to a single sensor can destroy the entire jet.
Subcontractors All the Way Down
I wrote in January that Boeing is a prime example of a company whose production-chain is subject to “O-ring” vulnerabilities. Why? In order for an aircraft to operate safely many critical components must work together more-or-less perfectly—the margin of error is tiny. This means that “droppers” pose a heightened risk.
Further, aircraft are technologically sophisticated products with long, complicated production chains. Because of the non-linear dynamics present in O-ring vulnerabilities, adding more steps to an already long production chain greatly increases the risk of failure.
Enter the Indian subcontractors.
Mark Rabin, a former Boeing software engineer, told Bloomberg that hiring Indian subcontractors “was far less efficient than Boeing engineers just writing the code” as “it took many rounds going back and forth because the code [written by the subcontractors] was not done correctly.”
Violà, the Indian subcontractors were the “droppers.”
Even if we assume that the Indian subcontractors were just as competent as their American counterparts, hiring them was a mistake simply because they added an extra link to the production chain. And as we know, this increased the risk of overall failure in a nonlinear way.
Unfortunately, passengers paid the ultimate price for Boeing’s obsession with complexity. I say obsession because this is not the first instance of Boeing being bitten by complex production chains.
Boeing’s 787 Dreamliner also suffered onerous delays, cost-overruns, and safety problems. Why? Boeing outsourced the design and production of the aircraft to some 50 different contractors, and the company likewise subcontracted the planes’ production.
In the end, God only knows how many different subcontractors, located in how many different countries, contributed to the aircraft.
Boeing in effect was building a puzzle in the shape of an aircraft, without any way of knowing if there were any “droppers” until it was too late. This is the economic equivalent of the Hindu’s infinite regression—it’s subcontractors all the way down.
O-ring vulnerabilities are not just Boeing’s problem: they’re America’s problem. Why? The supply chains of most American corporations are so entangled with foreign producers that at this point even small disruptions or mistakes in remote corners of the world could bring leviathans of industry—like Boeing or Apple—to their knees. And with them, America itself.
America’s Founders were well-aware of the risks that complexity and foreign suppliers posed, and it is part of the reason that America’s first substantive piece of legislation was the Tariff Act of 1789. If America is to remain prosperous and free, we must simplify our economy by removing unnecessary steps—by firing our Indian, Chinese, and Mexican subcontractors.
We need to bring our factories home.
Photo Credit: Mark Wilson/Getty Images