737s parked
6 min read

Non-pilot acquaintances who know of my aviation background often ask about the Boeing 737 Max and its MCAS system. I do my best to explain what the MCAS system is designed to do, and how it works.

I tell them MCAS moves the pitch trim to add force to the controls under some conditions of airspeed, CG and other factors. I also explain that if MCAS runs amuck, as it possibly did in the two fatal accidents involving the 737 Max, the situation is no different than a trim runaway, a failure we pilots have always been trained to handle.

Many are amazed that the MCAS system doesn’t have multiple backup modes to detect failures and disable the system. I say that’s what the pilots are for. The pilots are the backup to the automated system in airplanes of the certification vintage of the 737.

Boeing 737

To recover from a system failure, first you have to recognize it.

When the pitch trim system runs away, the pilot has multiple methods of disabling it. The first line of defense is a button under the pilot’s thumb. There are also switches, and in most airplanes, a circuit breaker than can be pulled to totally depower the trim system.

This information doesn’t sit well with most non-pilots. It seems like too complicated a task for people to reliably perform every time.

What really blows away the non-pilot is when I explain how critical systems such as autopilots and pitch trim are certified. The rules are that test pilots must recognize the trim failure, for example, by some positive event such as seeing the trim wheel moving un-commanded, or sensing an unexpected force on the flight controls.

Once that positive identification is made, the test pilot must wait exactly three seconds before taking the proper action to disable to system. The FAA calls those three seconds “recognition time.” During the final stages of a landing approach, the recognition time is reduced to 1.5 seconds in the belief the pilot is paying closer attention during that critical phase of flight.

When non-pilots hear this information, their mouths drop open. They utter something like “that’s crazy. Three seconds! That’s just nuts.”

In my career of flying with so many test pilots, and writing about certification programs and standards, the three seconds sounded totally logical to me. Count them out in your head. Three seconds is lots of time to get the thumb of your flying hand on a button on the control wheel.

And this belief in the rationality of the recognition time was reinforced every year or more during simulator training. At least one pitch trim runaway is part of every initial or recurrent simulator training program. Sim instructors often send the trim off on a run during high workload periods such as when configuring for landing when the flaps and gear extension are changing the stick force. The more diabolical instructors will introduce a trim runaway during those brief periods of cruise flight in the sim when the autopilot is flying and your reading a checklist or studying the procedure for the next approach. Identifying a trim runaway in straight and level cruise with everything working fine can be the most difficult.

Even though the Level D sims jet pilots train in faithfully reproduce the behavior and flying qualities of the actual airplane, sim training is not the real world of flying. In the sim we are all primed at every moment for something to fail, because it usually does. Over the years I’ve learned to keep my right knee pushed against the trim wheel on airplanes that have a wheel, such as the Citation family, so I feel the trim runaway moving the wheel instantly. That works great in the sim, but in real life flying do I do that? Of course not. Nobody does.

And that’s the root of the myth of the perfect pilot. During training and checking, we all must meet a standard that is essentially perfect. In the case of a trim runway, we must demonstrate without fail that we do in fact recognize the failure during the three-second period, and react properly to recover.

In the past, the perfect pilot was essential for development and certification of airplanes. It has been either technically impossible, or too costly, to include enough monitors to automatically detect and handle the failure of critical systems such as pitch trim. The pilots were the backup, both as monitor, and then as a “system” to recover from the failure.

737s parked

Is the answer better pilots or more fly-by-wire?

It turns out that pilots have been mostly perfect. Trim failures do happen, but in nearly all situations, the crew reacts correctly and the flight continues to a safe landing. Of course there have been fatal accidents caused by trim failures, and not just a few, but they are small in number compared to the total number of flights.

But at the jet level, flying has become so safe, that even a small number of accidents deemed acceptable even 25 years ago, is no longer tolerable. And electronic advances make it possible to include at least triple-redundant monitoring and sensing in critical flight control systems. Fly-by-wire (FBW) control systems—which all newer designed larger jets employ—use triple redundant computers and sensors to operate all flight controls, including trim. Pilots’ movement of the cockpit controls are the equivalent of moving the mouse on your desktop computer.

If you believe that the public has more confidence in a well-trained and experienced human pilot than in a FBW automated control system, try explaining the three-second recognition time. Once they understand that human pilots have three seconds to recognize pending disaster, and take the proper action to prevent a crash every time without fail, the public wants the multiple backups of FBW computers.

If you don’t believe me, read and listen to coverage of the 737 Max situation. The very idea that human pilots must be relied on to disable a malfunctioning critical flight control system stuns the media and, according to polls, the flying public. It doesn’t matter that the vast fleet of 737s—the most common jet airliner in the world—flies on with a pitch trim system that can run away, and that pilots must recognize and respond correctly to prevent a crash. That fact hasn’t been brought to public attention. The three-second recognition time is never mentioned, or even understood. If it was, the media and public would be as alarmed as they are about the MCAS system on the Max. They trust pilots, but unlike many of us who are pilots, the public knows there is no perfect pilot.

Mac McClellan
35 replies
  1. CharleyA
    CharleyA says:

    Public expectations have been driven by the technology revolution and its attendant hype. Reality is that technologies don’t always work the way people expect, writ Teslas wrecking, cellular phones failing inside dense structures, no Wifi everywhere….. People still need to think and sometimes act.

  2. Mark
    Mark says:

    I get that the real world is different from the sim, and I also get that the way in which MCAS can (or could) grab control of the trim system presents a bit differently than the runway trim scenarios practiced in the simulator. But I just can’t get past the fact that the data shows on both accident aircraft that the pilots experienced uncommanded trim movement not just once, not even just twice, but MULTIPLE times. You would think after the trim wheel moves a couple of times uncommanded even the most dense pilot would get the hint there is something drastically wrong with the electric trim system. This is even more true for the Ethiopian Air crew—led by a 9000 hour captain—who were purportedly briefed on the emergency AD issued describing MCAS and how to defeat it.

    It doesn’t require forgiveness of Boeing’s poor engineering decisions to recognize there was a serious deficiency with the training these two crews received.

  3. Ethan Levi
    Ethan Levi says:

    While a perfect pilot is a myth, an airplane’s design, its description in the Aircraft Flight Manual (AFM), and its simulator training sure can help an imperfect pilot recognize and fix a problem. Unfortunately, Boeing’s AFM did not inform pilots about the MCAS system, how it worked, or how to counteract it. Sadly the simulator was also incapable of realistically reproducing the force needed to overcome MCAS. If MCAS had been designed not to function on a single point of failure (AOA disagreement), if the AFM had a description of MCAS, and had the pilots been trained on realistic MCAS failure modes that would have helped pilots save airplanes and lives.

  4. YARS (Tom Yarsley)
    YARS (Tom Yarsley) says:

    Mac’s earlier column, “Can Boeing trust pilots,” was a masterpiece. With all of the public uproar, the question is becoming (already has become?) “SHOULD Boeing trust pilots?” Should any of us? Great question; consequential answers.

  5. Adam
    Adam says:

    Well written.
    While there are no perfect pilots even the training now is vital to get the pilot flying to recognize the issue even if outside the sim it may take more than three seconds. Start building the muscle memory to start the right action to recover while announcing the issue so the pilot managing can work the checklist. In the end this justifies not going away from a two pilot cockpit. The force needed to hold the yoke or overcome what was happening without training a second pilot was needed to work on the solution. Safety comes from redundancy. Multiple sensors multiple pilots.

  6. YARS (Tom Yarsley)
    YARS (Tom Yarsley) says:

    Safety does not come from redundancy. Safety comes from reliability.
    Redundancy is just one strategy for attempting to provide reliability.
    It’s not a distinction without a difference.

  7. Bruce Erwin
    Bruce Erwin says:

    Looking past the romance and prestige associated with being an airline pilot, each fly by wire advancement is one step closer to several airliners, then without windshields but with nose cameras, being simultaneously remotely flown as drones by centralized on-the-ground SIM-like pilots. It’s inevitable this concept resides in the minds of airline CEO’s. Whether the public will accept riding on pilotless airliners is another topic.

  8. Chris Barker
    Chris Barker says:

    There are no perfect pilots, and the ones who consider themselves perfect (or even really good) are perhaps the ones who will come a cropper when the trim goes amok. More importantly, as you hinted at in your piece, if the trim misbehaves at a critical phase of flight when you should be concentrating on your instruments and the weather is bad, your ability to deal with it will be reduced. Or what about if you’re really tired, or if you’re already dealing with another emergency?

  9. Chuck
    Chuck says:

    Whoa, hold on there…….

    I’m an airline pilot and flown all the Boeing’s from the B-52 to the Max plus a 15 year diversion to the MD-80 and in my 40 years of simulator training, I’ve never been given a runaway trim situation. Keep in mind that over the last five or so years the FAA mandated train to proficiency
    syllabus, the trading scenarios are strictly mandated with no instructor deviation.

  10. Joel Godston
    Joel Godston says:

    Oh My! I was trained to become a pilot in the Air Force. The training to receive your wings, to become a pilot took about 2 years with training, soloing, etc, in Piper Cub, T-28, and then in a T-33 aircraft. I flew B-47 aircraft in SAC…. then T-33, F-84, anf F-86H aircraft in the Mass, ANG, and finally owned and flew a Cessna 182. I have flown for 50+ years; but no longer fly at the age of 85! In my opinion there is NOTHING like training a REAL airplane. I understand the training to become a pilot these days is with simulators….very little training to become a pilot training in REAL airplanes. That IS CRAZY. Amen!

  11. Bill Babis
    Bill Babis says:

    The greatest safety device in any aircraft is a well trained pilot that understands safety is an attitude. Take that out of the flight equation and there is a 100% chance that safety is compromised.

  12. John Marshall
    John Marshall says:

    Mac has it right. I don’t know where Chuck flies, but I have a very similar background, and I have had several stab trim runaway scenarios in various simulators. Boeing airplanes are all very similar, the memory item for a stab trim runaway is the same, and can be accomplished in much less that three seconds. The elephant in the room with the Max is that both accidents occurred at third world airlines with crews trained accordingly. Neither first officer had more than 500 hours. In the Ethiopian crash the thrust levers were left in the takeoff detent for the entire flight: HELLO? Political correctness and politics have unfortunately become major players in this entire unfortunate situation.

    • Stephen Leonard
      Stephen Leonard says:

      Amen to John Marshall’s observation. I’ve never had a trim runaway in a real airplane (though I have in simulators, and I’m not even an airline pilot), but I’ve spoken to a number who have, in the MAX, earlier 737s, and various others, and the description is always the same. They pantomime the thumb-press followed by hitting the stab trim switch on the pedestal, which takes far less time to perform than to type here, followed by “now fly the airplane.” The jumpseat pilot the night before the Lion Air crash knew that, too.

      As for the public — here as everywhere, including our political life, they are at the mercy of ignorant but opinionated reporters who publish all manner of blithering nonsense, leading to near-hysteria and emotion-driven policy decisions.

      • Marc Rodstein
        Marc Rodstein says:

        “As for the public — here as everywhere, including our political life, they are at the mercy of ignorant but opinionated reporters who publish all manner of blithering nonsense, leading to near-hysteria and emotion-driven policy decisions.”

        The “ignorant but opinionated reporters” MAKE THEIR LIVING off of hysteria. The media no longer counts truth as a goal. The only thing that matters is increasing circulation or viewer count.

  13. Bob Hamilton
    Bob Hamilton says:

    I am a Traffic Safety Engineer and we use a similar “recognition” time in our analyses. Usually 2.4 seconds is assigned to any abrupt event requiring recognition, decision, and action. This applies to cross traffic, braking, animals or pedestrians, bicycles, etc. It’s a long time and is a great argument for driving slower.
    Direct parallel to the 3 second test pilot rule!!

    • Mac McClellan
      Mac McClellan says:

      Interesting on the 2.4 seconds in traffic situations, Bob. I don’t know the root of the three second recognition time in airplane certification but it was there when I began writing about avionics and aircraft certification in the 1970s. There may be some “science” behind the number, but it may also be a tradition that nobody knows or remembers where it came from and why.
      Mac Mc

  14. BobM
    BobM says:

    Lion Air continued to operate an unairworthy aircraft on multiple flights for at least 3 days prior to the fatal crash. Their maintenance attempts to fix the problem were mostly ineffective & finally made the situation much worse with a faulty installation of an AOA. On the two final flights, the stick shaker activated at take off rotation & was continuous for the rest of the flight, yet in both cases, the crew retracted flaps at low altitude. No competent crew would do this which increases stall speed 30-40 kts. Prudent airmanship & common sense would require leaving flaps down & return for landingg which means MCAS would never activate. On the final, fatal flight, the capt. countered MCAS with the thumb switch more than 20 times, yet never activated the trim cut out switches which is the final step of the IMMEDIATE ACTION memory item checklist for runaway trim which has been basic to every 737 for more than 50 yrs. The Ethiopian crash is even more inexplicable after 5 months of worldwide publicity over Lion Air, the FAA AD, & Boeing advisories which thoroughly explained MCAS & how to handle failures which cause it to activate. Yet this crew basically became “deer in the headlights” & crashed even quicker.
    In hindsight, Boeing should have implemented MCAS in a more failure tolerant way, but how do you design a jet airplane to be safe in the hands of an operator which lacks a robust safety culture & flown by crews who are inadequately trained & lacking in basic airmanship?

    • Mac MCCLELLAN
      Mac MCCLELLAN says:

      Good points all, BobM. Not that long ago most people would have concerns about flying on airlines from the developing world. But the global airline safety record has become so good that those concerns seem to have vanished. What remains is a demand that Boeing and Airbus build airplanes that are perfectly safe no matter how minimally trained the crew may be, or how deficient the safety culture of the airline operating it.
      So, the certification concept that the crew will respond correctly every time to an emergency is essentially dead. The only solution I can see–and the one being hammered on relentlessly in the media and even Congress–is more automation that can prevent and emergency, and then handle the problem if one does occur.
      Mac Mc

      • RichM
        RichM says:

        The only perfectly safe airplane is one that stays on the ground!

        As a flight control and flying qualities engineer for over 35 years, I am a bit perplexed as to why Boeing chose to implement MCAS with a single source of AOA. There are multiple probes as well as inertially derived AOA for inline fault monitoring.

  15. Steve Green
    Steve Green says:

    I have to disagree with the argument that an MCAS failure is no different than a runaway stabilizer. This is indeed the premise behind Boeing’s original safety analysis, but I believe it is simply wrong.

    For most of aviation history, the FAA and manufacturers have assumed that the probability of the pilot complying with published procedures is exactly one…in other words, they will execute the procedure without fail. They did this because they had no meaningful way of estimating the probabilities of a pilot failing to do what he/she is supposed to do. Alas, the perfect pilot needs half a chance to start with…

    The first line of defense in a trim runaway, for Boeing transports, is actually not under the pilot’s thumb. It is the control column cutout switches under the floor. The first step of the runaway stabilizer procedure is “Control Column – Hold Firmly”. If you do that, the column force opposes the direction of trim motion and the trim motor is cut out. The MCAS bypasses this feature, as it would obviously have to in order to achieve its purpose. However, the fact that the control column cutout switches are present on every Boeing since the 707, and are bypassed by the MCAS, illustrates the flaw in Boeing’s safety analysis. The runaway stabilizer procedure does not adequately address the MCAS malfunction, because the very first step in the procedure is moot.

    And if the control column cutout switches were not necessary, they would not be there. They are very likely installed to meet the three second requirement of FAR 25.255. The first recognition will be of unusual pitching movement, but at that point you don’t know whether it is the autopilot, pitch trim or something else. The 737 speed trim system will run the pitch trim frequently during the early climbout, so uncommanded trim motion alone is not an immediate clue. The obvious first response is to resist the pitching movement. Bear in mind that despite everyone’s claims, there are no “memory” items in the runaway stabilizer procedure, unless the operator so specifies (mine does not; we have zero “memory” items since we transitioned to a Quick Reference Card system). We could debate whether there should be…and all of us, including the Ethiopian crew, understand the reality of immediately disabling the pitch trim with the master trim cutout switches on the center pedestal.

    Next, in his book “Handling the Big Jets”, UK CAA test pilot Dave Davies clearly explains the threats created by a runaway stabilizer, and he also specifically states that in such a case, it is vital to “get the speed off”. Sadly, the latter point did not make it to the Ethiopian crew’s training, but frankly, no one has ever mentioned it to me in my 35+ years of airline flying, either. It should be intuitive, but not necessarily if you are task saturated with low total experience while the stick shaker is banging away.

    Davies also talks about the technique for manual trimming a heavily loaded stabilizer, the “roller-coaster” maneuver, in which you have to unload the stabilizer for brief periods to manually reposition it. He mentions that both pilots may have to work the manual trim crank together. These ideas are vaguely reflected in the Boeing 737 Flight Crew Training Manual, but appear nowhere in the 737 QRH or FCOM. They are clearly relevant, but in our zeal to modernize, simplify and reduce training to “performance based” criteria, we just dumped them. In my experience on the 737, they have never been discussed, published or trained.

    I have never had a runaway stabilizer event simulated during 737 training. We had a spate of real runaways in the MD80, and that led to simulator training. At various points in my career with different airlines, we have done simulated runaway stabilizers on the 757/767, which is a different animal because there is no manual trim wheel nor any immediate indication of trim motion; hence the control column switches are essential. We must have done runaways on the 727, but that was a long time ago.

    What the public really needs to be shocked about is how the perfect pilot is crippled by poor safety assessments and lack of critical information even before leaving the gate.

    • Steve Green
      Steve Green says:

      I should have clarified…there are no LONGER any memory items. Years ago, there were Boeing recommended recall tems for the runaway stabilizer, but even before we did away with all memory items, we had done away with memory items for the runaway stabilizer. They were retained on the MD80.

    • Bobm
      Bobm says:

      The original Boeing procedure, from 737-200 50 yrs ago, for runaway nose down trim was an immediate action memory item. Pull back against the nose down trim while SIMULTANEOUSLY trimming nose up with the thumb switch, then cut the trim switches & trim manually. This works whether the failure is MCAS or something else. If the procedure is no longer taught in this way & as a memory item, that is very dangerous & may be what killed Lion Air. The captain countered MCAS with the thumb switch more than 20 times over a period of some 10 minutes, but never hit the cutout switches. He then turned control over to the 1st officer while he consulted the QRH. The Muslim 1st officer said a prayer to Allah & did little else while MCAS trimmed full nose down.The MAX accidents are a stark reminder that runaway trim is not to be trifled with & must be nipped in the bud.
      What makes no sense to me is why any of these 3 crews retracted flaps with an active stick shaker. It simply defies logic & shows a serious lack of situational awareness & airmanship.

    • Brett Garner
      Brett Garner says:

      This is a great answer and should make us all cognisant that out superior us training may not have saved us either.

  16. Jack Morris
    Jack Morris says:

    The best automation in airplanes cannot replace a pilot with the skill and experience necessary to recognize a problem and apply the corrective action to mitigate or eliminate the problem. I can think of two recent examples that underscore this:

    1. Air France Flight 477 where the crew (due to instrument malfunction) did not recognize that the Airbus 330 had entered a stall and according to other experienced pilots who examined the post flight data, the crew could have stabilized the aircraft through a simple adjustment in engine power to a prescribed setting. See PBS Nova episode: Crash of Flight 477.

    2. Qantas Flight 32 in which an jet engine disk failure caused significant damage to hydraulic lines critical to flight control systems on an Airbus 380 with triple redundant systems. A book, QF32, by the pilot, Richard de Crespigny, describes this harrowing incident and it is clear that his experience and outstanding capabilities led to a safe outcome rather than what would likely have been a disastrous crash. I highly recommend this book for anyone wanting a better understanding of the issues of Fly By Wire aircraft operated by sophisticated computer systems.

    Although these systems do add to the safety of flight, it is an illusion to think that they are perfect and when something really goes wrong, it is often the pilot who makes the difference between life or death.

    Also, it is easy to be critical of pilots (or crews in the era of CRM) who make a mistake. All we can do is hope they have the necessary training, skill and experience, and maybe some sort of right stuff that can’t be quantified but makes a difference when things go wrong.

  17. Mike Hogan
    Mike Hogan says:

    Your reference to Fly-by-wire (FBW) control systems, [which] use triple redundant computers and sensors to operate all flight controls, doesn’t apply to the current MCAS design which used only one of the two Angle of Attack sensors. According to AW&ST reports, Boeing’s MCAS fix incorporates a voting system that monitors inputs from both AoA sensors and, if they disagree, disables MCAS. This is revision is still not a “triple redundant” design. The planes would need a third AoA sensor to do that. All that said, I have wondered why all the attention as been focused on MCAS fixes and not on review/correction of the AoA sensors, which were the root cause of the MCAS trim reponse that brought the planes down. Seems to me an analysis of the design/construction of the AoA sensors is warranted here: how often do they fail, how resistant are they to insets/debris, are maintenance procedures adequate (apparently not in the case of the Lion Air crash).

    • Steve Green
      Steve Green says:

      Mike, one of the most salient questions remaining unanswered is: how did three separate AoA vanes fail on two virtually new airplanes within six months? A corollary question is, why did the AoA vane replacement on the Lion Air flight not correct the problem?

      There is a great deal that we still don’t know, and, sadly, may never know, depending on the quality of the reports generated by the respective authorities.

  18. YARS (Tom Yarsley)
    YARS (Tom Yarsley) says:

    “The Muslim 1st officer said a prayer to Allah & did little else while MCAS trimmed full nose down.”

    “All we can do is hope they have the necessary training, skill and experience, and maybe some sort of right stuff that can’t be quantified but makes a difference when things go wrong.”

    So, we’re left with hope and prayers? How about if some of these third world “pilots” learn how to fly an airplane? Seriously.

  19. Pete
    Pete says:

    A well known flight training company has the mantra ‘the best safety device in an aircraft is a well trained crew’ – the inverse meaning of this is that the crew is the most dangerous system on an aircraft. It’s why crew training is so important.
    But any experience in flight test and in-service support at an airplane company teaches you that pilots do not follow checklists – humans are not machines and are guaranteed to make mistakes, they rebel against procedures and generally do things wrong on a daily basis. This is known and understood, as a result systems must be designed with this consideration in mind.
    For Boeing to have designed an automatic primary flight control system (e.g. controlling pitch) with a single-tier redundancy and dependancy on crew intervention as a safeguard, well somebody must have been subject to undue financial pressure with unchallenged engineering authority to let that happen. Everybody’s a damn fool for 15 minutes a day – including pilots! This has to be acknowledged in system design, in addition to ‘recognition time’, ARP4754 compliance notwithstanding! Dependancy on skill of the pilot – the most fallible and easily saturated system on an aircraft, must be treated with extreme caution.
    Meanwhile, system engineers should be made to undertake mandatory air experience flying so that for instance they can ‘feel’ what trim forces feel like at the extreme end of travel!

  20. YARS (Tom Yarsley)
    YARS (Tom Yarsley) says:

    “Dependancy on skill of the pilot – the most fallible and easily saturated system on an aircraft, must be treated with extreme caution.”

    Best endorsement I’ve read, for autonomous aircraft – no human intervention allowed at all.

  21. Vince
    Vince says:

    Interesting to read the calls for “autonomous aircraft.” This, in the near term is a pipe dream. I presently work as a Captain for a Major US airline, and have flown Airbus, Boeing and McDonnell-Douglas aircraft. All aircraft have automation glitches, and minor ones happen with a fair amount of regularity — and this is mature and certified technology. YARS, have you ever been in the control seat of an A320 when an autoland starts going terribly wrong at about 50′? I have, and I’m glad for one that I was.

    • YARS (Tom Yarsley)
      YARS (Tom Yarsley) says:

      Most pilots think that an autonomous aircraft control system is an “autopilot” on steroids. It is nothing of the sort, and has very little in common with autopilots or FMS devices.

      I owe AvWeb an article on the topic.

  22. Enderson Rafael
    Enderson Rafael says:

    Precise as an ILS cat 3B. Very nice article, Mac! Yes, the romantic era is gone for good, and, after over a couple years on the legendary 737NG – that many forget, is the safest airliner of all times – I must say that, if not as fun, flying the fly by wire 787 is much easier. The stab trim problem becomes a walk in the park. And you were so right: with a loss rate equivalent to a DC10, an A300 or even the classic versions of the 737 itself, the MAX was grounded. We just can’t afford going backwards in the numbers, which is a good thing of course. And although there is obviously room for improvement on the MCAS (and it must be done), and we all Boeing pilots felt a bit betrayed when first heard of the MCAS (that yes, should be on the FCOM, why not? The autoslat function was always there, and we will spend our careers without seeing it coming to play), the decision to ground the MAX was much more than technical – and in that sense, it would have been commercially much wiser from Boeing and FAA to be the first to do it. But what is done is done. Now is time for maybe, recognize that we became victims of how reliable acft had become, and sad enough, in the romantic way, that includes getting a bit less of us on the final safety equation.

Comments are closed.