A lot happened in 2019 from a technology perspective. Google built a 53 qubit quantum computer that could generate 10 quadrillion numbers. SpaceX made rockets flying backwards seem normal. And Facebook was able to bring together Democrats and Republicans; though it was to criticize their privacy policies. Sadly the most important tech event of 2019 is the death of 346 passengers due to a software bug.
The Boeing 737 is one of the most successful planes in history with 10,000 built over 50 years. Not surprisingly Boeing has been careful to make new 737 models fly like older ones so that pilots could easily take control without training. However when Boeing added a larger engine to the 737 Max to improve fuel efficiency the flight characteristics changed in a fatal way.
The larger engine size required Boeing engineers to mount the engine slightly forward and up on the wing to maintain ground clearance. Unfortunately the new positioning caused the plane to pitch up under power. To counter the upward basis Boeing developed a software package called the Maneuvering Characteristics Augmentation System (MCAS) to tilt the plane down.
In 2019 two 737 Max’s crashed when MCAS took control of the plane and pushed them into the ground. The situation brings back the horror in 2001 Space Odyssey of HAL deciding to get rid of Dave except with a lot more people. While it is uncertain when the Max will fly again what is certain is that no pilot will ever allow MCAS to take control of the plane.
Perhaps the most troubling aspect of the 737 Max crash was that Boeing’s flight control software team is the best in the world. Boeing’s software development process spec’s out every line of code and module. All software modules are carefully designed and tested independently and then re-tested as a system. The entire system has a backup. There is also a back up system to the back up system. If the elite Boeing team can’t manage the software complexity of their plane what hope is there for rest of us overworked developers?
With software now responsible for protecting human life whether in the form of flight control software, autonomous vehicle navigation or smart city management it’s critical these systems operate according to design. Regulators and Congress must implement common sense policies that ensure safety is embedded into high complexity software systems. Here are three ideas to consider:
Human Override Must Be Mandatory
Human override must be designed into every life impacting system incase there is a software failure. It’s also critical that the override system not be dependent on the same software system that is failing. A pilot or driver should never have to fight a computer. The simple act of putting a hand on the wheel should be enough to take control.
Systems Must Be Verified By Independent Testers
MCAS was developed so that pilots would not have to go thru re-certification or re-training to fly the new 737 Max. Unfortunately the 737 Max was flight tested by experts in MCAS. In a logical quandary unless a pilot had expertise in MCAS software they can’t really fly the plane. Systems should be tested by independent personnel who are neither employees nor contractors to verify safe operation.
New Software = New Product
Visually the 737 Max looks like its older versions but it’s a different plane. Whenever a like impacting system gets new control software it must be treated as a new product whether it’s a plane or autonomous vehicle. Updated systems should be re-tested to verify safe operation including the human override function.
The next decade will take software complexity to new heights as quantum computers, space-based communications and connected systems manage the planet we live on. It’s very tempting to say, “machine-learning is the solution to complexity”. While machine-learning systems are great at building statistical models of known events they produce random results when given a new situation. We’re going to have a lot of new situations in the coming decade!
For more on these topics:
We published a list of the most popular stories on OODA loop for 2019 here.
We dive deeper into AI and what it means for business and our collective future in our special series on AI, which includes:
- AI Security: Four Things to Focus on Right Now – This is the only security framework we have seen that helps prevent AI issues before they develop
- A Decision-Maker’s Guide to Artificial Intelligence – This plain english overview will give you the insights you need to drive corporate decisions
- When Artificial Intelligence Goes Wrong – By studying issues we can help mitigate them
- Artificial Intelligence for Business Advantage – The reason we use AI in business is to accomplish goals. Here are best practices for doing just that
- The Future of AI Policy is Largely Unwritten – Congressman Will Hurd provides insight on the emerging technologies of AI and Machine Learning.
- AI Will Test American Values In The Battlefield – How will military leaders deal with AI that may treat troops as expendable assets to win the “game”.
- The AI Capabilities DoD Says They Need The Most – Savvy businesses will pay attention to what this major customer wants.
