Having read your book QF32 and a number of resources available online, please let me ask one question regarding your video interview on youtube.
During the interview, you talk about the decision whether to follow the ECAM instructions to switch off the ENG 4 hydraulic pumps (“5 and 6″), and how you suggested that “everybody think about this for some seconds” before pressing the guarded pushbutton, to figure out if it makes sense to reduce redundancy on the still working yellow hydraulic system.
The journalists asks you whether, in hindsight, you have found an explanation as to why ECAM would prompt you to switch those pumps off, and your answer is somewhat along the lines of, you were not allowed to disclose those details at that time, but it would be in the final report, and we should be surprised when reading the answer.
Of course, I have also read the final report, but obviously I missed the respective answer (it is quite an overwhelming read; I cannot even imagine what kind of experience that must have been for you in the real-life situation).
So I would kindly like to ask, whether you could today share a few technical insights into that question (i.e. if it is legal for you today). Why was it sensible to switch off the ENG 4 HYD pumps? 😉
Thank you very much,
Mike from Germany
Hi Mike from Germany,
Thanks for your question. I am now able to discuss two critical pieces of information that I was not able to include in my book.
Airline flying is a very professional business that is unforgiving of mistakes.
The A380’s hydraulic system is called “2H + 2E”, meaning that is consists of:
- 2 Hydraulic Systems (Green, Yellow) (conventional hydraulics) (1.2 tons lighter than triple hydraulic systems)
- 2 Electro-Hydraulic Systems (for critical flight applications)
The A380 has two independent hydraulic systems named “Green” and “Yellow”. (Interestingly these colours were inherited from the Concorde over 50 years ago, that had three independent hydraulic systems named Green, Blue and Yellow, but that is another story).
- The 120 litre Green hydraulic reservoir is located in pylon 1, above Engine 1 on the left wing. The Green hydraulics are powered by a total of four engine driven pumps on engines 1 & 2. The green system contains 585 litres of Skydrol hydraulic fluid.
- The 120 litre Yellow hydraulic reservoir is located in pylon 4, above Engine 4 on the right wing. The Yellow hydraulics are powered by a total of four our engine driven pumps on engines 3 & 4. The yellow system contains 545 litres of Skydrol hydraulic fluid.
Skydrol (or an alternate fluid called “HyJet IV”) is a specially coloured hydraulic fluid that is authorised to be used in the A380. Skydrol is dangerous. It is so dry and acidic that it burns flesh-eyes. Any hydraulic leaks in the A380’s 5,000 psi system is extremely dangerous as the stream of high velocity oil acts as a knife capable of cutting through flesh, bones and thin metal. The good thing is that this fluid self ignites at a very high temperature of about 507 degrees Celsius (engine oil self ignites at about 280 deg C, so you might find it interesting to re-read QF32 at page 320). Skydrol costs $25 per litre!
2.0 The Hydraulic Situation on board QF32
The hydraulic problems we faced on QF32 are described at QF32 page 200.
Engine number 2 had exploded. The ECAM checklists instructed us to shut down the Green hydraulics. It then told us to shut down half of the Yellow hydraulics.
Matt’s hand reached up towards the overhead panel to disconnect the drive shaft for the fifth and sixth pumps out of the total of eight pumps in the Green and Yellow hydraulics. Pumps 5 & 6 were located on Engine 4. If Matt pressed the disconnect push button for this engine, then the pumps would not have been recoverable and we would be left with just 2 engine driven hydraulic pumps on Engine 3 to power the entire aircraft. (The A380 also has electrical hydraulic pumps at some flight controls but some of these were also inoperable.)
2.1 What we did
“Stop! Can we all please think about this for ten seconds?” I called this to stop Matt irreversibly shutting down hydraulic pump s that might affect our destiny without first giving the ECAM and our situation more thought.
My reasoning was obvious. Why are we shutting down hydraulic pumps on Engine 4 located at the extreme of the right wing when engine 2 exploded on the left wing? It did not make sense!
I used a rapid decision making method to poll the crew for their thoughts, but the decision what to do with pumps 5 and 6 would be mine. I was responsible for the lives of 469 passenger and crew.
2.2 Why we did it
I decided to have the two engine driven hydraulic pumps at engine 4 disconnected because:
- Perhaps metal filings or other contaminants or other problems had been detected in the engine 4 hydraulic pumps that might spread and damage the two pumps on engine 3.
- The engine 3 hydraulic pumps must be protected at all costs.
Matt completed the ECAM actions. This left Nancy-Bird Walton with just two engine driven hydraulic pumps and a couple of small electrical pumps at a few controls.
3.0 What we later found out about the hydraulics
The ATSB report on QF32 had not been released when I wrote my book “QF32”. Federal laws prohibited me from discussing the following until the ATSB report was published. Even when it was released, this high level “consumer friendly” ATSB report omitted lots of detailed and fascinating information. I can now explain two surprising facts.
3.1 The Green hydraulics reservoir was full
This is fascinating! All the pilots that day saw the ECAM checklists and watched the hydraulic System Display show the bleed air pressure in the green hydraulic reservoir (to stop pump cavitation) fail, then the hydraulic fluid level reduce to zero and then the Green Hydraulic system fail.
We discovered six months after the flight that the green reservoir was in fact full at the end of the flight, suggesting that the indications we saw and the ECAM warnings were wrong, and that perhaps we might not have needed to turn off the hydraulics on Engine 1.
3.2 The Yellow hydraulics system was operating normally
We did not need to disconnect the two engine driven hydraulic pumps on engine 4. Again, the severed wires and quadrax cables limited our view, the ECAM’s assessment and our understanding of the aircraft’s status. Engine 4 had degraded to an ALTerNate mode with its maximum thrust reduced as a consequence. The severed-shorting wires (communications and logic) probably reduced the ECAM’s ability to understand the Yellow hydraulic system.
The ECAM checklist instructed us to disconnect the engine driven hydraulic pumps on Engine 4. I initiated a discussion amongst the pilots whether we should follow ECAM’s suggested actions.
I ultimately decided that we did not know the status of the aircraft as thoroughly as ECAM did. So in this case, with acceptable hydraulic reserves remaining, I decided that we should follow the ECAM procedures and disconnect pumps five and six.
Our logic was that disconnecting hydraulic pumps five and six would protect the last two remaining hydraulic pumps on engine 3.
Today I still think that our logic that we applied on the 4th November 2010 was correct, based upon what we experienced and deduced.
3.3 Why the confusion?
There is no explanation for this confusing and contradictory information other than my and the ATSB’s guesses that some of the 650 wires and Quadrax (4 aluminium wire duplicated twisted pair ) network cables were severed and shorting. This meant that incorrect, reduced or no hydraulics information was delivered to the independent Hydraulic System Monitoring computers, ECAM’s flight warning computers, the overhead panels and schematic displays. (We also received reduced or incorrect information about the brakes, fuel, engines and many other systems.)
I am not criticising aviation certification standards, Airbus nor the A380. Aviation learns through failures and we are fortunate to be able to analyse these failures with the intention to improve safety.
3.3.1 Very technical
Rolls-Royce Trent 900 engines use the CAN bus for many communications to the A380’s Input/Output Modules (IOM).
These simple two wire twisted pair protocols might not be resilient to hacking or sensor and open circuit failures. For example, earthed automobile engine oil pressure transducers connect via just one single wire to the oil pressure indicator. These instruments cannot differential between an open circuit and a zero. So a car’s oil pressure indicator will incorrectly display zero oil pressure when the engine is running if the wire is disconnected from the pressure transducer.
More secure, advanced and resilient communication (such as Quadrax cables in an Airbus AFDX network) are used for critical communications. These complex systems offer additional resilience to errors such as open circuit and sensor identification-presence, failures and hacking. Unfortunately many of these quadrax cables were severed during the QF32 incident.
4. Apollo 13 – Sensor Failure!
Let’s solve the problem, team … let’s not make it any worse by guessing”
We were in a mind space during QF32 after the explosion that was similar to the mood of the controllers at Houston during the Apollo 13 mission. The NASA controllers were making no progress after after the oxygen tank exploded.
The controllers were living in a “fog of war”. Virtually every controller had problems and no one could see a pattern in it all. Gene Kranz (the lead Flight Director for Apollo 13) said it was like “living a bad dream”.
Gene told his team “Let’s solve the problem, team … let’s not make it any worse by guessing”.
The controllers looked at two of Apollo 13’s failing power systems. They initially thought they had lost the instrument readings due to a high gain antenna alignment problem. Eventually, with his team’s help, Glynn Lunney (one of the four NASA Flight Directors on duty) made sense of the parameters.
Failure was not an option
Lunney decided to permanently shut the reactant valves in two fuel cells to preserve oxygen for the third fuel cell. It was a courageous decision and the best decision in the circumstances, and similar to our decision on QF32 to disconnect hydraulic pumps 5 and 6 to protect pumps 7 & 8.
In our case the remaining hydraulic pumps on Engine 3 worked faultlessly. However in Apollo 13’s case the oxygen leak and confusion continued. Commander Jim Lovell then faced a total loss of the electrical and oxygen systems …
Gene’s (Tiger) team had to quickly decide how to return Apollo 13 to Earth. Failure was not an option. They had the choice of a high-risk fast (U turn) abort that could get the astronauts home in 34 hours, or to conduct a rocket burn to rejoin the free return trajectory that would get them home two days later. They were 20 hours short of electrical power and 36 hours short in water. The fast return assumed the main engine would still function, something that Gene correctly resisted.
Gene’s low risk and conservative option turned out to be the correct choice and a great exercise in crisis management and decision making. With the help of his team, Gene made the courageous decision just 57 minutes after the oxygen tank exploded to commit to the longer trip (around the moon) home.
5. Back to QF32
There was confusion with QF32. QF32’s aircraft communications, networks and computers were so damaged and disrupted that we could never have expected to fully appraise the hydraulics systems from the ECAM messages, the overhead panel displays, nor from the System Displays (that each normally use independent sensors).
Computers and automation: A case of “Pearls in – Pearls out”, or “Garbage in – Garbage out”?
Faced with the deluge of warning messages being received at Mascot, the engineers on the ground thought that the ECAM had failed and that it needed to be reset. But unlike Apollo 13, QF32 had lost its satellite voice communications. Any decisions we had to make on the aircraft, we had to make on our own.
Mike, if you felt overwhelmed when reading QF32, then I can assure you that the pilots were also operating at their limits in the cockpit that day. We were also living in our “fog of war”.
Solve the problem but don’t make it worse by assuming, presuming or guessing
I wrote in my book QF32 how the fuel system overwhelmed me, which is why I prepared for the Armstrong Spiral to mitigate a loss of all engines. The ATSB investigators later told me that the Fuel Quantity Management System computers were so compromised that ECAM would never have made sense of the the fuel systems nor give us valid guidance about the fuel systems on QF32 that day.
The hydraulics systems however were a bit easier to understand than the fuel systems, and a lot more manageable.
6. Did we do the right actions?
I am proud of the knowledge reasoning and calm that the crew exercised on QF32 when we faced the confusing sets of ECAM checklists, overhead panels and systems displays.
Airline flying is a very professional business that is unforgiving of mistakes. Our handling of this ECAM checklist was an example of how we delayed actioning an ECAM checklist until we had analysed the system, the procedure and its consequences – whether it made sense and whether it was the right thing to do. This was an example of why pilots must have a deep knowledge and understanding of their aircraft’s systems.
Ignorance is never an excuse. Know your machine inside out.
With the power of hindsight, the actions that we took five years ago still make sense today (based on what we faced in the cockpit back then). None of this information that we received months after the flight makes me think that our logic was wrong on the day when we disconnected the hydraulic pumps for engines 1 and 4.
I am always keen to receive advice and learn from others, and happy to admit if I have made a mistake. There are many armchair quarterback critics that have different views to mine and who criticised our actions. However if I was faced with the same black swan event today with the failures, displays and ECAMs today, that we had on QF32, then I would do the same actions again today.
Mike I hope that this long reply answers your question.
I have equally complex answers to the questions of why we did not land quickly after the explosion, and why we chose not to order the passengers down the evacuation slides after landing. These discussions are all food for thought.
QF32 was an example of team excellence, where 8 teams pooled their knowledge, training and experience working to survive a black swan event.
- I am proud of the pilots (Matt Hicks, Dave Evans, Harry Wubben and Mark Johnson).
- I am proud of CSM Michael Von Reth and his cabin crew
I think QF32 was a “successful failure” because all 469 passengers and crew were returned home to their loved ones without injury after a black swan event. It was also a “successful failure” for the lessons learned.
Risk is the price of progress and discovery. We have to have the courage to accept risk and to continually push against the boundaries of science if we are to grow and develop to become a wiser and kinder manager of our planet. We also need the knowledge, experience and teamwork to conquer the unknown.
Finally, we need inspired leaders at the leading edge like Neil Armstrong, Jim Lovell and Eugene Kranz who could identify, rate and work with risk and prove resilient in the Fog of Battle.
By definition, you cannot train for a black swan event. However you can prepare for the unexpected. Resilient people anticipate failures and understand how systems fail. Because prevention is impossible. So you will have to mitigate the failures and use your knowledge and teamwork to counter-attack. Because when you have the confidence to make the courageous decisions, that’s when you will become intrepid leaders of intrepid teams, and that’s when you will become bullet proof and not gun shy and best able to survive the “unknown unknowns”.
I have the highest respect for my airline, Airbus, Rolls-Royce and the A380.
- I have the highest respect and confidence for the Rolls Royce Trent 900 engine.
- I consider the A380 is still the biggest most comfortable, most quiet and greatest passenger jet aircraft in the sky.
Come fly with us and we’ll show you WHY!
A380 closing @ 1,800 kilometres per hour 4,000 feet above (RDC)