Technical Lessons from QF32

With my suggestions for the Industry

Wonder (Painting by my friend Coplu (Coplu.Com))

Wonder (Painting by my friend Coplu (Coplu.Com))

Jaanus Torp writes:

A good picture of an uncontained turbine failure

A good picture of an uncontained turbine failure

Hello Richard,

I just finished your book and I might say that it is not suitable for before bed reading.  Even though I had read the final report and knew the outcome, I was still quite wired by the masterfully written gripping drama.

I have a question about matters that you might not want to discuss or have not enough information about but which were left in the air.

The ATSB’s Final Report on QF32 focuses mainly on recommendations to Rolls-Royce and touches only very lightly on Airbus’s involvement. Can you please give some insight into what Airbus learned from this flight and did they make any changes to the design and/or operations of the A380 systems/layout/automation?

Cheers,
Jaanus

Richard replies:

Hi Jaanus, thanks for your question.

Airbus learned a great deal from the QF32 incident.

RIP the world's best friend - Neil Armstrong, designer of the Armstrong Spiral and who said "expect the unexpected" (RDC)

RIP – Neil Armstrong, designer of the Armstrong Spiral (RDC)

I visited Airbus, Toulouse many times after the QF32 event.  I  delivered presentations about the QF32 event to the RAeS, engineers, test pilots and Airbus’ global leaders and executives.   We discussed Airbus philosophies, methods, practices and how aircraft operations might be improved.

I thanked Airbus for designing and building the A380, a remarkably resilient aircraft that in my case returned 469 passengers and crew home to their loved ones, and saved over 32,000 friends the grief that comes to others in the event of a disaster.  (a future story)

Here is a summary of my discussions and sugestions-predictions for change.

1. Philosophies:

  1. (Photo QF32 Passenger Kjell Ljungqvist)

    Photo taken by QF32 Passenger Kjell Ljungqvist

    Fly-By-Wire (FBW) flight controls, autothrust and flight management automation all  worked well and as expected. (no change)

  2. Install a “LAND ASAP” push button on the ECAM control panel, that inverts the logic (QF32 p198) and gives the pilots the minimum essential instructions to land/ditch immediately.   (R&D)
    (See another discussion on this topic)
  3. Install my implementation of Neil Armstrong’s Spiral (QF32 p170)  into future Flight Management Systems.  (unlikely, due to accompanying airline costs to train the manoeuvre)

2. Practices:

  1. The SOPs, role and task divisions for two pilot operations worked well.   (no change)
  2. Include in pilots’ manuals a detailed description about flight control checks in fly by wire aircraft.  (commenced)
  3. Pilot training to include Stress Free Deliberate Practice   (hopeful)

3. Aircraft:

  1. Damaged lift on QF32's wing tips (Photo Richard de Crespigny)

    Damaged to lift at QF32’s wing tips to streamlining ailerons (Photo Richard de Crespigny)

    Mount more cameras inside and outside the aircraft for pilots to view the cabin, cargo holds, electronics bays, wheels and engines from within the cockpit.  (unknown)

  2. Modify ECAM displays to show the total number of pending ECAM checklists (i.e. 2 of 10).  (accomplished on the A350)
  3. Selectively and progressively detune the attention-getting alarms if an ECAM checklist is in progress.  This will reduce pilot distraction.  (unknown)
  4. Very technical: Improve some of the the sensors and the CAN busses for OEM systems to better discriminate between an open circuit and a zero.   (unknown)
  5. I think some of the ECAM checklists produced incorrect logic (FUEL, HYD …) because of item 4 above.   This was a case of garbage (sensors) -in-garbage (logic) out.   The fuel systems were so confused by the sensor, pump,valves and duct failures  that the Fuel Quantity Management System (FQMS) computers probably needed an alternate program (like the flight controls “Alternate Law”).   I think the unusual ECAM behaviour on board QF32 was unavoidable under the circumstances, and an example of why pilots are still needed to resolve the unexpected.  (human controllers still required in High Reliability Organisations, See also:  Sully Sullenberger: Technology Cannot Replace Pilots)
  6. A380 (Trent 900) fan blades, showing the wide chord, forward and aft sweep, and extensive washout. (Photo: RDC)

    A380 (Trent 900) fan blades, showing the wide chord, forward and aft sweep, and extensive washout. (Photo: RDC)

    Technical:  Include an ECAM advisory message (like that on the B787) to advise that the flight controls are saturated (pilots should make slow and deliberate inputs).   (unknown)

  7. It was bad luck that the wires to the brakes and brake temperature sensors on the left wing were cut.  The system displays and ECAM could never have fully resolved the brakes’ situation (see item 5 above) .  (no recommendation-change)
  8. Very Technical:  Include an ECAM advisory message to warn that the flight controls are out of phase with the pilot’s inputs (potentially-dangerously inducing rate-limited pilot induced oscillations).

Note: The damage to QF32, wing and fuselage was considerably worse than the damage reported at page 38 of this 2010 report by the AIA  on High Bypass Ratio Turbine Engine Uncontained Rotor Events.

4. Comments

Upper left wing - QF32 (Photo Ulf M. Waschbusch)

Upper left wing, showing the effects of the significant vortices that are created upwind of the damage then spread-trail back to the trailing edge. (Photo Ulf M. Waschbusch)

4.1 Aerodynamic Damage

_MG_2773 (1000x563)

Matthew Orchard, Head of Design Wing (ESW) with the remarkable A380 wing at the Airbus Wing Factory, Broughton (Photo: Airbus)

We discussed the:

  1. damage to the wing, the flight controls, and its effects on the aircraft’s controllability,
  2. incorrect performance calculations that produced approach speeds that were too slow and that gave insufficient margin to the stall (QF32 p259),
  3. incorrect flight displays that resulted because of the aerodynamic damage to the wing, and the
  4. “SPEED SPEED” and “STALL STALL” warnings that we heard during the approach when we slowed down 1 knot below out approach speed.
Frank Ogilvie, father of the A380 at the Airbus Wing Factory (Photo: Airbus)

Frank Ogilvie (former Aerodynamics Director and Deputy Head of Overall Design for the A380 project) in front of the bottom panel of the A380 wing at the Airbus Wing Factory, Broughton, UK.  (Photo: Airbus)

Technical:

  1. Simulation of upper surface exposed to battle damage (T.W. Pickhaver & P.M. Render RAeS Journal Aug2105 p937)

    Simulation of upper surface exposed to battle damage (T.W. Pickhaver & P.M. Render RAeS Journal Aug2105 p937)

    Many people have tried, but there is no easy way to predict the aerodynamic effects of damage to an aircraft’s wing.   In the case where there is a hole passing through top and bottom wing panels, wind tunnel tests (see image to the right) show a pair of horse-shoe vortices starting upstream from the hole, then broadening downstream on both sides to the trailing edge.  The flow separates behind the damage and there is significant reverse flow.  The lift and aerodynamic moments are significantly affected.

  2. Frank Ogilvie, father of the A380 at the Airbus Wing Factory (Photo: Airbus)

    Frank Ogilvie (former Aerodynamics Director and Deputy Head of Overall Design for the A380 project) at the Airbus Wing Factory, Broughton, UK (Photo: Airbus)

    Remember that damage to one wing must be replicated (in reduced lift) to the other wing if the aircraft is to fly straight.

  3. For aft loaded airfoils (such as all modern supercritical wings), their is a dramatic loss of lift when a trailing edge control surface (i.e. aileron) slipstreams.   (see the description after QF32 page 238)
  4. Damage effects are amplified when the damage is located inboard on modern transonic aircraft wings (that have a triangular lift distribution).
Airbus Wing Factory (Photo: Airbus)

Matthew Orchard, Head of Design Wing (ESW) shows me the A380 wing with landing gear attachment lugs.  Airbus Wing Factory, Broughton (Photo: Airbus)

In the event of wing damage, I think the practical solution for predicting the effects of the damage, determining the approach speeds and then flying the approach lies more with the pilot:

  1. knowing the key JAR-25/CS-25 aircraft certification standards and performance margins, and
  2. know the “what”, why,” “how”, and “if-thens” of controllability checks (specifically for fly-by-wire aircraft).
Head Up Display (Painting by Coplu (Coplu.Com))

Head-Up Display (Painting by Coplu (Coplu.Com))

4.2 Electrical  Damage

Part of electrical damage to the left wing

Part of electrical damage to the left wing

We discussed the extreme number (more than 650) of wires and network cables that were cut and the loss of systems that resulted from damage inside the left wing and the belly of the fuselage.  Even though Engine number 2 exploded, the damage extended to include Engine 1.  Four separate pairs of wires that took separate paths to two independent fuel shutoff valves for Engine 1 were all cut, rendering us unable to shut down Engine 1 until three and a half hours after we landed (QF32 p323).   Four pairs of wires to two fire extinguishers on Engine 1 were also cut, rendering the Engine 1 fire extinguishers useless.  (bad luck – no suggestion)

Electrical and forward wing spar damage to the left wing

Electrical and forward wing spar damage to the left wing

This was very bad luck for us though it displayed the resilience of the Rolls-Royce Trent 900 engines to still control thrust after so many wires had been cut.

Technical:   I asked an ATSB QF32 investigator how many more wires we could have lost and still made it home.  He said “none to Engine 1”.  I’ll leave it to you to work through that interesting scenario.

Many of the fabulous QF32 cabin crew.

Many of the fabulous QF32 cabin crew.

4.3 Airline Training

Many airlines have updated their training programs to incorporate lessons learned from the QF32 incident.  I believe cabin crew training at British Airways, Air France, Lufthansa, JAL, Singapore Airlines, Virgin Atlantic and Virgin Australia have included discussions about QF32.

Constellation - painted by one of the world's best aviation artists - Jaak De Koninck (www.jaakdekoninck.be)

Another wonderful painting by Jaak De Koninck (www.jaakdekoninck.be)

4.4  Aviation Organisations

I have presented the QF32 story and my thoughts to the World’s most specialised aviation organisations (regulators, safety, security, pilot organisations, investigators, manufacturers, suppliers, financiers, insurers and airlines).

Summary

Jaanus, I maintain absolute respect and admiration for Airbus and the manner in which the A380 was designed, built and tested.

A380 – it’s not just a passenger magnet, it’s also a pilot magnet.

Only the most critical operational changes are ever made to aircraft after they pass certification tests.

Any damned fool can criticize, but it takes a genius to design it in the first place (Edgar Schmued)

(Graph: RDC)

(Graph: RDC)

Airbus aircraft are all designed with a common strategic and operational philosophy that extends from the first A320 FBW aircraft to its latest A350.   A philosophical change to how one aircraft type operates would by definition need to be retrofitted to the entire fleet of more than 8,000 Airbus FBW aircraft!

I am not disappointed that Airbus will probably implement few of my recommend changes.  It’s a lot easier for me to think of quick and short term narrow fixes to individual aircraft designs, than it is for Airbus to design and integrate these changes throughout all of their aircraft fleets.  I am confident that the critical changes will eventually be implemented.

Thank you Gelly Kalouta and the Dubai Marriott Hotel for your scaled A380!

Thank you Gelly Kalouta and the JW Marriott Marquis, Dubai  for your scaled A380!

Nancy-Bird suffered over 500 fuselage impacts from shrapnel. I wrote that the probability of this incident happening again is one in 10^-14, that is, one million times less probable than the most stringent certification standards.   The aircraft flew remarkably well, which is a testament to the Airbus designers, builders, testers and maintainers.

When I have the privilege to fly Nancy-Bird, I tell my passengers before the flight that they are lucky to be travelling on an aircraft that is dear to my heart. Nancy-Bird has been stress tested and case hardened more than any other aircraft in the sky – and it proved itself indestructible that day.

I provide the feedback and lessons above to serve as tools for your toolbox of solutions that you might need one day when you face the unknown unknowns.  Every incident is different.  Every incident has it’s own unique threats, risks and outcomes.  So these “Technical Lessons from QF32” should only viewed in their context as another case study and food for thought.

The Passenger and Pilot Magnet

Don’t let appearances deceive you! This is a $400m and 4m piece Passenger and Pilot Magnet

To every Airbus employee, thank you again for designing and building such a resilient aircraft.

It is my life’s ambition to be a pilot, and my pleasure to fly the A380.  Come fly with me and I’ll show you why.

The A380 – it’s not just a passenger magnet, it’s also a pilot magnet.

(Photo RDC)

(Photo RDC)

See also:

Discussions:

  • Evacuation vs Deplane with steps
  • Airborne vs Land ASAP  (future)
  • Aircraft Flight Control Checks  (the “what”, why,” “how”, and “if-thens”)  (future)
  • Crisis management (my next book ….)
  • Leadership & building resilient teams (my next book ….)

5 comments

  1. […] Technical Lessons from QF32 […]

  2. Emily Cresswell · · Reply

    Hi Richard,

    I think you did an exellent good on QF32 and everyone else on board that flight that day. (Including Singapore ATC, Singapore police etc).

    There is one question I would like to ask:

    1). How comes you didn’t pass your route check?

    Thanks again,

    Emily

  3. […] Source: Technical Lessons from QF32 […]

  4. John Ellis · · Reply

    Question 1. Do you think there would be a safety benefit to having AFDX (Ethernet)/ARINC-664 connectivity to the engine sensors and control systems for the “Black Swan” type event?

    Question 2. Its unlikely, but possible that 4 separate ARINC-429 links are lost together in an emergency. As I understand, ARINC-429 link would loose functionality completely if the bus wires are cut, but AFDX virtual link could be switched around damaged physical links, without loss of functionality.

    Question 3. Was the loss of electrical power on the left wing as a result of a sensor issue, automatic shutdown, direct damage, or was the breaker pulled by the crew? Loss of electrical power would likely cause the backup electrohydrostatic actuators (EHAs) to loose functionality as well.

    Thank you Captain, I enjoyed the posts as always.

    1. Hi John,

      Answer 1 – Yes. Airbus owns the IP for AFDX. It is interesting that Wikipedia lists AFDX technologies being used in the: Airbus A380, A400M, A350; Boeing 787; Sukhoi Superjet 100; ATR 42 &amp ATR 72; AgustaWestland AW101, AW149, AW189 & AW169; Irkut MS-21; Bombardier Global Express & CSeries; Learjet 85; and Comac ARJ21.

      Note 1: Each RR Trent 900 engine has about 80-100 sensors. These feed at about a 5Hz rate to the Engine Monitoring Unit (EMU) at each engine (for Vibration and Engine health monitoring and for creating novelty reports for RR R&D). Some of these parameters are exported to other aircraft systems via CAN buses. (Each of the two Trent XWBs on the first test A350 (MSN1) had about 600 parameters).

      Note 2: The No. 2 engine EMU data was available for the flight duration, until 3.4 seconds after the engine failure. (QF32 report p193)

      Answer 2. – I agree

      Be careful though. Network resilience is the sum of network control logic, router resilience and cabling resilience. This is a complex topic covering CAN Bus, ARINC standard, AFDX, and Quadrax and other cables. I am not sure that I am the right person to provide these answers, but I’ll try:

      With all things considered, it would not surprise me if, over time, aircraft manufacturers and original equipment manufacturers (OEMs) move their secure and mission critical systems over to resilient AFDX architectures.

      An infrastructure comprising many trunked AFDX (Avionics Full DupleX switched ethernet) switches, AFDX software and Quadrax (twin twisted pair ethernet) cabling provides great redundancy and resilience . However these are more proprietary and costly technologies to deploy compared to the loose and simple alternative that is offered by the simple CAN bus.

      CAN BUS

      The CAN bus is a loose communications standard by Boch/ISO that is used extensively in automobiles. The CAN bus is also being used in aircraft for communications between off-the-shelf engines, buyer furnished equipment and avionics. The CAN bus is also used instead of the high speed (but low redundancy) ARINC 429 standard. Both of these standards are being replaced and extended by the better ARINC 825 aircraft standard.

      The CAN bus is neither resilient nor secure. For example, there is no facility to prevent installing additional nodes onto a CAN bus to spoof, intercept or interfere with communications. An additional physical device that is attached as a node in a CAN network can participate in all network communications. There are plenty of stories of hacking of cars’ CAN busses and taking control.

      Avionics are designed with basic software logic to control-limit how to respond to data received on the network. This is important for the “Chris Roberts” discussion below.

      To the aircraft manufacturers grief, some suppliers of third party Buyer Furnished Equipment (such as IFE systems) in the past have issued incorrect installation instructions to airlines that inadvertently breached the aircraft’s network topography and firewall protections. The aircraft manufacturers work with these OEMs to ensure that these problems are fixed.

      AFDX

      The best solution to counter the above flaws is an integrated and multilayered infrastructure comprising redundant AFDX software, AFDX routers and cables. This resilient architecture provides transparent redundancy, high bandwidth and secure communications. Airbus and Boeing aircraft implement two independent levels of AFDX to also protect them from systematic errors. (The Airbus PRIMary Flight Control Computers have three levels of protection).

      Wireless Technologies

      Wireless networks offer solutions to many of the above problems whilst introducing new problems. Wireless networks get around the problem of cut/intercepted/shortened wires (like we had on QF32). However they introduce the threats of power supply failure, denial of service and hacking.

      Chris Roberts

      I am very skeptical of Chris Roberts’ claims of hacking into and altering aircraft systems. His ability to hack into a CAN bus and “read” data does not imply (and there is no evidence) that he can “write” to another critical system and take command.

      Summary – Network Resilience & Security

      Since 2004, Airbus has employed a team of many hackers whose role is just to find, then fix security vulnerabilities in their systems. I am confident that the threats from attackers such as Chris Roberts have been identified and countered. However security is a zero sum game. We must remain vigilant because success today is no vaccine for tomorrow.

      With all things considered, it would not surprise me if, over time, aircraft manufacturers and original equipment manufacturers (OEMs) move their secure and mission critical systems over to resilient AFDX architectures.

      Answer 3 – QF32 Electrics

      We lost many electrics because the some of the 6 Generator & Ground Power Control Units (GGPCUs) isolated generators (and thus busses) after detecting: under voltage, earth leakage, overload, open circuits and the Engine Fire push button activation. Loss of these power busses did depower some of the EHAs.

      See also: QF32’s hydraulics and the Fog of War

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: