Simulators began to make their way into aviation soon after the historic first flight of the Wright Flyer in 1903. Simulators have since expanded into simulation environments and further into holistic virtual or synthetic training ecosystems. Airline studies have long concluded that experienced pilots could be trained entirely on a qualifying simulator. With a pilot shortage upon us, the role for flight simulators is even greater. According to a 2019 industry report, the overall flight simulation industry is projected to grow into a $7.7 billion industry by 2025.
Flight simulators are powerful training tools; will artificial intelligence open up new ways to use them?
There is an increasing presence of artificial intelligence (AI) and machine learning (ML) in our lives today. While neither are new concepts, the decreasing costs of compute infrastructure have contributed to this trend. Machine Learning is a branch of AI where the algorithm “learns” from given data. It actively looks for patterns without the need to tell it to look for something specific. In one of its branches, ML algorithms can find patterns without even being told what to look for. Not having to programmatically define the one (or few) things we are looking for opens up the opportunity to discover insights without limiting it. Given a data set or data stream, the algorithm attempts to discover all/any patterns it observes. Of course, in order to keep it focused, it is always advantageous to operate with a set of focus areas at any time.
There are some very positive and pertinent use cases for AI and ML. One of those is in the enrichment and extensibility of flight simulators. Traditionally, flight simulators have been self-contained hardware that allowed for training. Their primary use was in flight training. Then there came the time where their use extended into design. This was a serious shift. Where at one time simulators were used in lieu of aircraft to train pilots, now they could be used to design, develop and test the aircraft itself. After that came the phase where their use had extended into the design of the ambient environment and associated processes. As an example, air traffic management innovations and airport design all benefit from simulators.
We are now at the cusp where combining capable simulators with high-powered compute environments can enhance safety in aviation. Consider this—can flight simulator data tell us more about yet to be known opportunities that can improve airspace safety; or tell us more about how to prevent loss of control incidents; prevent communication lapses from turning into serious issues?
Extending the line of thought, can the power of ML detect patterns of human behavior in a simulator? What would these findings be worth in the realm of safety?
The answer is that these insights could be immensely valuable. Pilots are a diverse population and over their career, most pilots will potentially fly across diverse geographies, languages, and cultures. Insights could be used to optimize human behavior under varying circumstances. These insights could point to both strengths and weaknesses. It is known that behavior, judgment, and action patterns remain consistent whether in a simulator or in the real thing. If that is the case, then observing patterns in a simulator will allow for preemptively addressing them.
An NTSB report about a Gulfstream incident in Massachusetts found that habitual non-compliance to procedure had preceded the incident. The facts behind a 2016 mishap in Southeast Asia involving the actioned shutdown of the working engine indicate one of the pilots had demonstrated similar tendencies during simulator checks. He had on multiple occasions shown signs of confusion when workload was heavy and judgment had been in question. Facts around a 2010 ultralight crash in Tatham, Australia, highlight the fact that the pilot was a habitual risk-taker and his airmanship was routinely questionable. These are a few of the hundreds of incidents where human behavior patterns have routinely preceded incidents. Could additional insights have saved the situation?
Whether its unlearning a habitual golf swing and relearning an optimized one, or unlearning an incorrect habitual action on the flight deck, it takes insights and then improvement actions. Today these insights come as observations by an instructor sitting inside a sim along with pilots either during ab-initio or recurrent training. There is nothing wrong with this model. It’s just that the human mind can only sift through so much data at a time.
What if pilots could get valuable feedback without a human instructor sitting there?
Machine-driven contextual analysis of large volumes of data is another league in itself. Simulators are aptly positioned to collect operating data. ML has the compelling power to sort through the data, discover hidden patterns, and provide the audience with answers to questions that they may not have thought of asking. This means ML can crunch through thousands of hours of simulator data and come up with findings that we wouldn’t have known even to ask or look for. ML algorithms can crunch these numbers to recommend opportunities for improvement and help build customized training plans. Machine Learning can also recommend training improvements across cohorts of pilot population.
The next question would be to determine where this capability would best apply. Who would benefit most from such insights?
Insights at any level are always useful. Deep insights are of course even more useful. A good pilot is always learning. That said, in my opinion, general aviation stands to benefit the most. Most GA pilots do not operate within the rigorous regime of controlled operations and training. Furthermore, funding is scarce. I say “most GA pilots,” because there are some GA pilots who operate in environments similar to the operational regime of a routine air carrier—but this is more the exception. This is even more applicable those pilots for whom flying is not a profession. Recreational GA pilots go two years in between formal training events. Habits learned during training matter, but those that have been formed between checkrides matter even more. Given that, GA pilots will gain much from such insights and help correct any negative tendencies they may have formed.
Detecting patterns where pilots fail to notice an approach to stall, or perform improper stall recovery, can be valuable. Numerous cases of a stall on a base-to-final turn are found. Loss of control is a serious issue. These are patterns found in a certain class of pilots, often low-hour pilots. Improper weight and balance calculations is a pattern of thinking found in certain categories of pilots. Launching into marginal weather or weather at a distance assuming that will stay in place has led to numerous flights into known IMC. Flying a wide traffic pattern is a habitual issue for some. Flying an unstabilized approach, delayed reactions to failure annunciations, or to air traffic communications are all patterns. Extending full flaps too early, overshooting final course on an approach, or improper execution of a go-around are all patterns of human behavior.
These don’t occur as one-off instances. Some are a result of improper training, others are incorrect actions that have become habitual with time. Nevertheless, they compromise safety and will benefit from correction.
Machine learning can learn from usage and automatically adapt future training scenarios to hone in on improvement areas. If not automatically, it could recommend adequately to allow an instructor to focus on patterns that go undetected by the human observer. After all, training is not one size-fits-all thing. Once an ML model has “learned,” it takes very little time for it to detect anomalies in new data; and each new data set or data stream enriches the model. Detection of anomalous behavior then becomes increasingly better with time because the model gets increasingly richer. This means that after a model has stabilized, a few hours of flying by a pilot will enable the model to detect patterns, present findings, and make recommendations. A machine learning model can be tuned for specific focus objectives such as a precise landing, or in general terms for overall pilot behavior which could comprise all/any aspects of pilot technique.
Simulators have the wonderful ability to collect data. They have always had it. However, we have never before had the luxury of having storage and compute infrastructure available at price points as they are now. The promise of machine learning is more real now than ever before. Simulators will be used more than ever before to train the pilots of tomorrow. The intersection of these two domains presents a compelling opportunity to use cutting edge insights and improve safety.
- Simulators: focus on saving time, not logging time - May 16, 2022
- Wish fulfilled: flying to Kitty Hawk - February 3, 2021
- Flight simulators, safety, and the power of AI - April 9, 2020
While not targeting the lighter side of GA, simulation companies such as FlightSafety (and CAE) are working on technology that incorporates AI to improve training. This is especially important where the instructor can’t always perceive exactly what the student is doing.
https://www.google.com/amp/s/www.ainonline.com/aviation-news/business-aviation/2019-11-11/flightsafety-debuts-smart-new-way-train%3Famp
Very interesting article. I’m curious to see how synthetic data will impact aerial AI. At Simerse, we’re working on a bleeding edge tool that may be of interest: https://www.simerse.com/