Technical – QF32’s Hydraulics, Apollo 13 and the “Fog of War”

Your highness Ms Universe by Coplu (Coplu.Com)

Your Highness Ms Universe by Coplu (Coplu.Com)

Mike asks:

Hi Richard,

Having read your book QF32 and a number of resources available online, please let me ask one question regarding  your video interview on youtube.

During the interview, you talk about the decision whether to follow the ECAM instructions to switch off the ENG 4 hydraulic pumps (“5 and 6″), and how you suggested that “everybody think about this for some seconds” before pressing the guarded pushbutton, to figure out if it makes sense to reduce redundancy on the still working yellow hydraulic system.

A380 Overhead Hydraulics Panel lit up in

A380 Overhead Hydraulics Panel lit up in “test mode”.  Faults are displayed in amber font.  Push button actions are displayed in white font  (Photo RDC)

The journalists asks you whether, in hindsight, you have found an explanation as to why ECAM would prompt you to switch those pumps off, and your answer is somewhat along the lines of, you were not allowed to disclose those details at that time, but it would be in the final report, and we should be surprised when reading the answer.

Of course, I have also read the final report, but obviously I missed the respective answer (it is quite an overwhelming read; I cannot even imagine what kind of experience that must have been for you in the real-life situation).

So I would kindly like to ask, whether you could today share a few technical insights into that question (i.e. if it is legal for you today). Why was it sensible to switch off the ENG 4 HYD pumps?  😉

Thank you very much,
Mike from Germany

Richard Answers

Hi Mike from Germany,

Thanks for your question. I am now able to discuss two critical pieces of information that I was not able to include in my book.

Airline flying is a very professional business that is unforgiving of mistakes.

1. Background

The A380’s hydraulic system is called “2H + 2E”, meaning that is consists of:

  • 2 Hydraulic Systems (Green, Yellow) (conventional hydraulics) (1.2 tons lighter than triple hydraulic systems)
  • 2 Electro-Hydraulic Systems (for critical flight applications)
Andrew Eccles in the Concorde simulator at Brooklands, UK (Photo RDC)

Andrew Eccles in the Concorde simulator at Brooklands, UK (Photo RDC)

The A380 has two independent hydraulic systems named “Green” and “Yellow”.  (Interestingly these colours were inherited from the Concorde over 50 years ago, that had three independent hydraulic systems named Green, Blue and Yellow, but that is another story).

  • The 120 litre Green hydraulic reservoir is located in pylon 1, above Engine 1 on the left wing.  The Green hydraulics are powered by a total of four engine driven pumps on engines 1 & 2.   The green system contains 585 litres of Skydrol hydraulic fluid.
  • The 120 litre Yellow hydraulic reservoir is located in pylon 4, above Engine 4 on the right wing.  The Yellow hydraulics are powered by a total of four our engine driven pumps on engines 3 & 4. The yellow system contains 545 litres of Skydrol hydraulic fluid.
A380 Overhead Hydraulics Panel during normal flight (Photo RDC)

A380 Overhead Hydraulics Panel during normal flight (Photo RDC)

Skydrol  (or an alternate fluid called “HyJet IV”) is a specially coloured hydraulic fluid that is authorised to be used in the A380.  Skydrol is dangerous.  It is so dry and acidic that it burns flesh-eyes.  Any hydraulic leaks in the A380’s 5,000 psi system is extremely dangerous as the stream of high velocity oil acts as a knife capable of cutting through flesh, bones and thin metal.   The good thing is that this fluid self ignites at a very high temperature of about 507 degrees Celsius (engine oil self ignites at about 280 deg C, so you might find it interesting to re-read QF32 at page 320).  Skydrol costs $25 per litre!

Oshkosh 2015

Oshkosh 2015

2.0  The Hydraulic Situation on board QF32

The hydraulic problems we faced on QF32 are described at QF32 page 200.

Engine number 2 had exploded.  The ECAM checklists instructed us to shut down the Green hydraulics.  It then told us to shut down half of the Yellow hydraulics.

A380 Hydraulic layout (Airbus)

Example A380 Hydraulic layout (Airbus)

Matt’s hand reached up towards the overhead panel to disconnect the drive shaft for the fifth and sixth pumps out of the total of eight pumps in the Green and Yellow hydraulics.   Pumps 5 & 6  were located on Engine 4.  If Matt pressed the disconnect push button for this engine, then the pumps would not have been recoverable and we would be left with just 2 engine driven hydraulic pumps on Engine 3 to power the entire aircraft.   (The A380 also has electrical hydraulic pumps at some flight controls but some of these were also inoperable.)

A380 Overhead Panel lit up in

A380 Overhead Panel lit up in “test mode” (Photo RDC)

2.1 What we did

Uncontained turbine failure - QF32

Uncontained turbine failure – QF32

“Stop!   Can we all please think about this for ten seconds?”   I called this to stop Matt irreversibly shutting down hydraulic pump s that might affect our destiny without first giving the ECAM and our situation more thought.

My reasoning was obvious.   Why are we shutting down hydraulic pumps on Engine 4 located at the extreme of the right wing when engine 2 exploded on the left wing?   It did not make sense!

I used a rapid decision making method to poll the crew for their thoughts, but the decision what to do with pumps 5 and 6 would be mine.   I was responsible for the lives of 469 passenger and crew.

2.2 Why we did it

I decided to have the two engine driven hydraulic pumps at engine 4 disconnected because:

  • Perhaps metal filings or other contaminants or other problems had been detected in the engine 4 hydraulic pumps that might spread and damage the two pumps on engine 3.
  • The engine 3 hydraulic pumps must be protected at all costs.

Matt completed the ECAM actions.   This left Nancy-Bird Walton with just two engine driven hydraulic pumps and a couple of small electrical pumps at a few controls.

3.0  What we later found out about the hydraulics

The ATSB report on QF32 had not been released when I wrote my book “QF32”.    Federal laws prohibited me from discussing the following until the ATSB report was published.  Even when it was released, this high level “consumer friendly” ATSB report omitted lots of detailed and fascinating information.  I can now explain two surprising facts.

3.1  The Green hydraulics reservoir was full

Hydraulics - How does the Sequoia tree (world's largest tree) pump water to the canopy 300' above the forest floor? Is there 8 atm of water pressure at the roots? Is the 300' tree height limited due to hydraulic pressure?

Hydraulics – How does the Sequoia tree (world’s largest tree) pump water to the canopy 300′ above the forest floor? Is there 8 atm (120 psi) of water pressure at the roots? Is the 300′ tree height limited due to hydraulic pressure?  (Physics for the Coffee Table)

This is fascinating!  All the pilots that day saw the ECAM checklists and watched the hydraulic System Display show the bleed air pressure in the green hydraulic reservoir (to stop pump cavitation) fail, then the hydraulic fluid level reduce to zero and then the Green Hydraulic system fail.

We discovered six months after the flight that the green reservoir was in fact full at the end of the flight, suggesting that the indications we saw and the ECAM warnings were wrong, and that perhaps we might not have needed to turn off the hydraulics on Engine 1.

3.2  The Yellow hydraulics system was operating normally

We did not need to disconnect the two engine driven hydraulic pumps on engine 4.   Again, the severed wires and quadrax cables limited our view, the ECAM’s assessment and our understanding of the aircraft’s status.   Engine 4 had degraded to an ALTerNate mode with its maximum thrust reduced as a consequence.   The severed-shorting wires (communications and logic)  probably reduced the ECAM’s ability to understand the Yellow hydraulic system.

Coral asks - are there extraordinary hydraulic pressures inside this large Sequoia tree (Sequoia National Forest, CA, USA)

Coral asks – are there extraordinary hydraulic pressures inside this large Sequoia tree (Sequoia National Forest, CA, USA)

The ECAM checklist instructed us to disconnect the engine driven hydraulic pumps on Engine 4.  I initiated a discussion amongst the pilots whether we should follow ECAM’s suggested  actions.

I ultimately decided that we did not know the status of the aircraft as thoroughly as ECAM did. So in this case, with acceptable hydraulic reserves remaining, I decided that we should follow the ECAM procedures and disconnect pumps five and six.

Our logic was that disconnecting hydraulic pumps five and six would protect the last two remaining hydraulic pumps on engine 3.

Today I still think that our logic that we applied on the 4th November 2010 was correct, based upon what we experienced and deduced.

Cut wires in leading edge - left wing

Damaged forward wing spar & fuel tank – left wing

3.3 Why the confusion?

There is no explanation for this confusing and contradictory information other than my and the ATSB’s guesses that some of the 650 wires and Quadrax (4 aluminium wire duplicated twisted pair ) network cables were severed and shorting.  This meant that incorrect, reduced or no hydraulics information was delivered to the independent Hydraulic System Monitoring computers, ECAM’s flight warning computers, the overhead panels and schematic displays.   (We also received reduced or incorrect information about the brakes, fuel, engines and many other systems.)

I am not criticising  aviation certification standards, Airbus nor the A380.   Aviation learns through failures and we are fortunate to be able to analyse these failures with the intention to improve safety.

3.3.1 Very technical

First stop on Jossie Harris' honeymoon - the A380 cockpit (VH-OQD) (26Aug15)

First stop on the Jossie Harris & Clint Berry honeymoon – the A380 cockpit (VH-OQD) (Dubai 27Aug15)

Aircraft and engine manufacturers sometimes use the automotive twisted pair CAN bus and ARINC 429 (an aviation extension of the CAN bus) protocols for insecure computer-sensor communications.

Rolls-Royce Trent 900 engines use the CAN bus for many communications to the A380’s Input/Output Modules (IOM).

These simple two wire twisted pair protocols might not be resilient to hacking or sensor and open circuit failures.  For example, earthed automobile engine oil pressure transducers connect via just one single wire to the oil pressure indicator.   These instruments cannot differential between an open circuit and a zero.   So a car’s oil pressure indicator will incorrectly display zero oil pressure when the engine is running if the wire is disconnected from the pressure transducer.

100s of cut belly wires, cables and air pipes

100s of cut belly wires, cables and air pipes

Damaged belly wiring and air pipes

Severed belly Quadrax cables and pipes

More secure, advanced and resilient communication (such as Quadrax cables in an Airbus AFDX network) are used for critical communications.  These complex systems offer additional resilience to errors such as open circuit and sensor identification-presence, failures and hacking.  Unfortunately many of these quadrax cables were severed during the QF32 incident.

4.   Apollo 13 – Sensor Failure!

Let’s solve the problem, team … let’s not make it any worse by guessing”

My honour to meet and interview the NASA legend Gene Kranz

My honour to meet and interview the NASA legend Gene Kranz (2015)

We were in a mind space during QF32 after the explosion that was similar to the mood of the controllers at Houston during the Apollo 13 mission.   The NASA controllers were making no progress after after the oxygen tank exploded.

The controllers were living in a “fog of war”.  Virtually every controller had problems and no one could see a pattern in it all.  Gene Kranz (the lead Flight Director for Apollo 13) said it was like “living a bad dream”.

Gene told his team “Let’s solve the problem, team … let’s not make it any worse by guessing”.

The controllers looked at two of Apollo 13’s failing power systems.  They initially thought they had lost the instrument readings due to a high gain antenna alignment problem.  Eventually, with his team’s help, Glynn Lunney  (one of the four NASA Flight Directors on duty) made sense of the parameters.

Failure was not an option

Lunney decided to permanently shut the reactant valves in two fuel cells to preserve oxygen for the third fuel cell.  It was a courageous decision and the best decision in the circumstances, and similar to our decision on QF32 to disconnect hydraulic pumps 5 and 6 to protect pumps 7 & 8.

In our case the remaining hydraulic pumps on Engine 3 worked faultlessly.  However in Apollo 13’s case the oxygen leak and confusion continued.  Commander Jim Lovell then faced a total loss of the electrical and oxygen systems  …

Commander Jim Lovell, Fred Haise, Bill Reeves (Apollo 13 Flight Controller) and Milt Windler (one of the four Apollo 13 Flight Directors)

Commander Jim Lovell, Fred Haise, Bill Reeves (Apollo 13 Flight Controller) and Milt Windler (one of the four Apollo 13 Flight Directors) at Oshkosh 2015

Gene’s (Tiger) team had to quickly decide how to return Apollo 13 to Earth.  Failure was not an option.  They had the choice of a high-risk fast (U turn) abort that could get the astronauts home in 34 hours, or to conduct a rocket burn to rejoin the free return trajectory that would get them home two days later.    They were 20 hours short of electrical power and 36 hours short in water.  The fast return assumed the main engine would still function, something that Gene correctly resisted.

Gene’s low risk and conservative option turned out to be the correct choice and a great exercise in crisis management and decision making.  With the help of his team, Gene made the courageous decision just 57 minutes after the oxygen tank exploded to commit to the longer trip (around the moon) home.

5.  Back to QF32

You are a lucky man Clint Berry! Congratulations (see Jossie above)

You are a lucky man Clint Berry! Congratulations (see Jossie above)

There was confusion with QF32.  QF32’s aircraft communications, networks and computers were so damaged and disrupted that we could never have expected to fully appraise the hydraulics systems from the ECAM messages, the overhead panel displays, nor from the System Displays (that each normally use independent sensors).

Computers and automation:  A case of “Pearls in – Pearls out”, or “Garbage in – Garbage out”?

Faced with the deluge of warning messages being received at Mascot, the engineers on the ground thought that the ECAM had failed and that it needed to be reset.    But unlike Apollo 13, QF32 had lost its satellite voice communications.   Any decisions we had to make on the aircraft, we had to make on our own.

Mike, if you felt overwhelmed when reading QF32, then I can assure you that the pilots were also operating at their limits in the cockpit that day.  We were also living in our “fog of war”.

Solve the problem but don’t make it worse by assuming, presuming or guessing

I wrote in my book QF32 how the fuel system overwhelmed me, which is why I prepared for the Armstrong Spiral to mitigate a loss of all engines.   The ATSB investigators later told me that the Fuel Quantity Management System computers were so compromised that ECAM would never have made sense of the the fuel systems nor give us valid guidance about the fuel systems on QF32 that day.

The hydraulics systems however were a bit easier to understand than the fuel systems, and a lot more manageable.

6.  Did we do the right actions?

I am proud of the knowledge reasoning and calm that the crew exercised on QF32 when we faced the confusing sets of ECAM checklists, overhead panels and systems displays.

Airline flying is a very professional business that is unforgiving of mistakes.  Our handling of this ECAM checklist was an example of how we delayed actioning an ECAM checklist until we had analysed the system, the procedure and its consequences – whether it made sense and whether it was the right thing to do.  This was an example of why pilots must have a deep knowledge and understanding of their aircraft’s systems.

Ignorance is never an excuse. Know your machine inside out.

With the power of hindsight, the actions that we took five years ago still make sense today (based on what we faced in the cockpit back then).    None of this information that we received months after the flight makes me think that our logic was wrong on the day when we disconnected the hydraulic pumps for engines 1 and 4.

I am always keen to receive advice and learn from others, and happy to admit if I have made a mistake.  There are many armchair quarterback critics that have different views to mine and who criticised our actions. However if I was faced with the same black swan event today with the failures, displays and ECAMs today, that we had on QF32, then I would do the same actions again today.

7.  Summary

Mike I hope that this long reply answers your question.

I have equally complex answers to the questions of why we did not land quickly after the explosion, and why we chose not to order the passengers down the evacuation slides after landing.  These discussions are all food for thought.

Karl Stefanovic with Dave, Harry, me and Mark taken during a 60 Minutes interview (Photo: 60 Minutes)

Karl Stefanovic with Dave, Harry, me, Matt and Mark taken during a 60 Minutes interview (Photo: 60 Minutes)

QF32 was an example of team excellence, where 8 teams pooled their knowledge, training and experience working to survive a black swan event.

  • I am proud of the pilots (Matt Hicks, Dave Evans, Harry Wubben and Mark Johnson).
  • I am proud of CSM Michael Von Reth and his cabin crew
Michael Von Reth

QF32 Customer Service Manager – Michael Von Reth

I think QF32 was a “successful failure” because all 469 passengers and crew were returned home to their loved ones without injury after a black swan event.  It was also a “successful failure” for the lessons learned.

Risk is the price of progress and discovery.  We have to have the courage to accept risk and to continually push against the boundaries of science if we are to grow and develop to become a wiser and kinder manager of our planet.  We also need the knowledge, experience and teamwork to conquer the unknown.

Finally, we need inspired leaders at the leading edge like Neil Armstrong, Jim Lovell and Eugene Kranz who could identify, rate and work with risk and prove resilient in the Fog of Battle.

Many of the fabulous QF32 cabin crew. Thank you!

Many of the fabulous QF32 cabin crew. Thank you!

By definition, you cannot train for a black swan event.  However you can prepare for the unexpected.  Resilient people anticipate failures and understand how systems fail.  Because prevention is impossible.  So you will have to mitigate the failures and use your knowledge and teamwork to counter-attack.  Because when you have the confidence to make the courageous decisions, that’s when you will become intrepid leaders of intrepid teams, and that’s when you will become bullet proof and not gun shy and best able to survive the “unknown unknowns”.

Some of the professional, competent and handsome cabin crew from QF32

Some of the professional, competent and handsome cabin crew from QF32

I have the highest respect for my airline, Airbus, Rolls-Royce and the A380.

  • I have the highest respect and confidence for the Rolls Royce Trent 900 engine.
  • I consider the A380 is still the biggest most comfortable, most quiet and greatest passenger jet aircraft in the sky.

Come fly with us and we’ll show you WHY!

Emirates crossing @ 1,800 kilometres per hour 4,000 feet above (RDC)

A380 closing @ 1,800 kilometres per hour 4,000 feet above (RDC)

See also

8 comments

  1. Dear Richard,

    Thank you very much for this detailed answer to my question! I have to apologize that I only checked back at your website a couple of months later and only now found the article, but I very much appreciate all the insights!

    Let me add to my initial post that I am a child of a – today retired – captain with Lufthansa who has, amongst others, flown on A330/340. I have to admit that back when I was a teenager, I often secretly read in his systems manuals because I was so fascinated of how aircraft systems were designed and operated. In fact, I was also a huge fan of the Apollo 13 movie, so I appreciate this side-story in your text, too 😀

    I guess that is also part of the reason why I was curious about the hydraulics situation and the ECAM instruction on pump shutdown on QF32. So even if life took me somewhere else but not into a jet cockpit, still today I think I do understand quite a bit of the details you presented in your article about the hydraulics architecture. And that is why I thank you so much for putting that much effort into the explanations!

    Greetings from Germany,

    Mike

  2. Robert Levine · · Reply

    Re: Apollo 13. The decision to shut the reactant valves on the second fuel cell (which was #1) was actually made by Glynn Lunney, who took over from Kranz as flight director about 90 minutes after the O2 tank blew. Kranz was, of course, lead flight director for the mission, but it’s interesting that Ken Mattingly (who was, of course, scheduled to be onboard until 2 days before lift-off) gives Lunney the lion’s share of the credit for sorting out the initial mess in his oral history interview for NASA.

    Kranz deserves a great deal of credit for the successful return of the crew, of course. But much of what has been written and filmed about Apollo 13 does underplay everyone else’s role.

    1. Thanks Robert,

      I agree with your comments and have updated the text.

      Gene Kranz writes about the events from page 306 of his book “Failure is not an Option”.

      All flight director teams dovetailed together as one enormous team (under Kranz’s leadership) to contribute to the $104b Apollo Program’s (1960-1973) objectives.

      NASA had a remarkable structure where specific commanders, astronauts and flight directors were each allocated to a specific Apollo mission. Each mission had specific goals to achieve on the singular path to get man on the moon by the end of that decade. Every one of the 400,000 people at NASA had a role to play in that strategy. I do not believe that there was any politics or tension between the teams – simply because everyone was incredibly busy planning their own mission and every team member relied on (and trusted) others to do their job.

      Gene Kranz was the chief of the Flight Control Division at NASA. Gene was also the Flight Director of the White team that was assigned as the Lead (Tiger) team for Apollo 13. Gene’s Tiger team compiled the mission plan for Apollo 13 then lead and executed the plan and liaise with the media. Kranz’s Tiger team was supported by three other teams under flight directors Gerry Griffin (Gold team), Frank Lunney (Black) and Milt Windler (Maroon) and the former flight director Chris Kraft.
      Gene and his White team  jacket.

      A win is a team win. All four flight directors and their teams worked the problems and returned Apollo 13 home. Gene is the first person to praise every Flight Director and team member for their extraordinary contribution to the Apollo Project.

      The leadership and teamwork that existed during the Apollo years serves as a role model other high reliability organisations. I am currently writing more about this in my next book.

  3. […] technical: Improve some of the the sensors and the CAN busses for OEM systems to better discriminate between an open circuit and a zero.   […]

  4. Jonathan Castelino · · Reply

    Hi Rich,

    Love the detail and clarity !

    On another note, I’m wondering if you ever got to the Udvar-Hazy aviation museum just a stones throw from KIAD. I had recommended a visit a couple of years ago. Quite a fascinating display, especially the space shuttle Discovery. I never knew how truly mammoth the shuttle was until I got close up to it at the museum.

    Cheers
    Jonathan

    1. Hi Jonathan. No I have not had the chance of getting to Washington state to see the Udvar-Hazy aviation museum. It’s on my to do list!

      The Museum of Flight at Seattle is an one of the best in the world. They might have a space shuttle trainer… Rich

  5. shripad kulkarni · · Reply

    very good…!!!!!!!!!!!!!!!!!

  6. What a great answer Richard!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: