System Design: Fixing the 787’s Batteries

 Boeing’s 787 Dreamliner is a complex design and massive engineering task that few people can truly appreciate. Not only is it an advanced composite structure aircraft with a loaded take-off weight of a half million pounds, but it was created in the light of modern fuel costs. It lists for at least $207 million a copy, so it is also one of earth’s most expensive machines.

The road to 787 completion involved the use of extensive composite airframe structures and close consultation with all vendors and subcontractors to minimize airframe weight burden in every sub-system and component. Each ship contains two large Lithium-ion batteries, one fore and one aft, so the weight burden in them had to be considered in the ship design.

It was those logical and completely understandable design decisions that eventually led to the two 61.8 pound Lithium-ion Cobalt Oxide battery failures, resulting in worldwide 787 fleet grounding while the problems of the unexpected battery failures were resolved.

Two 787s experienced dramatic battery failures, in two different locations on the airframe, less than two weeks apart. The first failure was on Jan. 7, on a grounded Japan Airlines (JAL) 787 in Boston; the second was on Jan. 16, on an All Nippon Airways (ANA) 787 in Japan. This led to FAA grounding of the airframe type in January by an emergency airworthiness directive, and other agencies worldwide followed shortly thereafter. All 50 of the planes in service were grounded, pending rectification of the battery problems, and approval of the solution.

The failed JAL 787 APU battery examined by the NTSB, with cell locations noted. Photo courtesy JAL

Reported “costs” of the world 787 fleet grounding vary considerably, and it is hard to quantify operator losses from inability to fly their newly purchased 787 fleets, but I believe it is an inarguably true statement to say that whatever possible weight and fuel savings might have been obtained by use of the Lithium-ion battery as opposed to the use of a more conventional NiCad or even SLA battery have long since been washed away in the correction costs and lost revenue. The announcement by Airbus that they would drop Lithium-ion batteries from its next major airframe, plus the residual uncertainty as to the actual root causes of the Lithium-ion battery pack failures in the two 787s to date, does not ease the minds of everyone, since no clear trigger cause was determined by either the National Transportation Safety Board (NTSB) or Japan Transport Safety Board (JTSB) in their investigations.

Understanding the Airframe

Most perceptions of the aircraft battery system come from much smaller aircraft, where it serves a variety of purposes from starting the engine to stabilizing the main DC bus impedance. The 787 is a much different electrical system concept, and could actually operate with no battery at all as long as ground power was available for starts. It has very high power generation capability, making a total of 1.5 megawatts in flight from 6 Variable Frequency Starter-Generators (VFSG), with two on each engine, and two in the aft Auxiliary Power Unit (APU).

Because the 787 is primarily an electrical, not a hydraulic or bleed air controlled aircraft, electrical power does almost every task on board the ship, requiring higher voltage to overcome distribution copper losses. These four busses are 235VAC primary power and 270VDC for pumps, plus 115VAC and 28VDC for small galley, IFE and flight deck subsystems. The ship relies on the low voltage DC 29.2VDC from the lithium battery only for engine starts if no ground power is available, for some power transfer functions and for last ditch back up emergency cockpit power. For it to actually be needed in flight, all six onboard generators would have to have failed, along with the emergency wind turbine.

Normal starts are from one of the external ground power ports, with the required energy supplied by an external ground power unit. Where a ground power start is not possible, the rear internal battery starts the APU, whose high capacity VFSGs then provide the power needed to start the two main engines forward. Both batteries (fore and aft) have now failed in service, however, making it more uncertain as to external causes of failure. The main battery is located just aft and below the cockpit area, The aft battery is just beyond the rear of the passenger compartment, in front of the APU.

Of some interest is that certification flight testing included a 5.5 hour failure simulation with one engine failed, and 5 of the 6 on board generators failed, and the 787 remained airworthy and operational. Its ability to work with large scale electrical failure is significant and impressive, despite its heavy reliance on electrical power.

The 787 LVP65 Battery Photo courtesy JTSB

The ship’s internal systems are powered by the four different voltage busses created by the six on-board generators, and the two batteries play no operational role there, except to start the engines (forward) or APU (aft) should no ground power be available, and to provide emergency pilot power for the cockpit. This is a radically different electrical bus structure than is found on most aircraft, and provides some insight into the motivation to keep the concept of a Lithium-ion battery system, even after the two battery fire incidents.

The primary need for the batteries in the 787 is for their high start (or cold cranking) current. The key attribute of the adopted Lithium-ion battery is a very high surge current rating of 1000A max., and 450A/45 seconds for APU starts, (plus slightly elevated DC voltage), roughly 4 times the surge current capability of the comparable earlier 767 Ni-Cad batteries according to Boeing, but with significantly less weight. The 8-cell GS Yuasa 787 LVP65-8-402 battery is rated at 75Ah and 29.6VDC, with total energy storage of 1440Wh, although the implied capacity is nominally 2200Wh from the parametric ratings.

Where things perhaps got off track for the 787 is that the lithium battery was then further optimized for low installed weight, which creates a significant problem when its primary start task is considered. Under the very high current start demands, there is a risk that the battery could overheat, deform, short, experience insulation failure or even catch fire as the high temperatures associated by the very high peak start currents pushed the lightened battery structure too far. These issues are especially significant in the Lithium-ion Cobalt Oxide chemistry picked for the 787 battery.

The GS Yuasa complete battery has a specific specification that no more than 3 APU start attempts can be performed at one instance, no doubt because of the very high thermal stress these operations create. The failure analysis of the battery system on the 787 foresaw a battery thermal failure event as being something so rare it could be expressed in terms of once in 10 million flight hours, but actually one failure occurred within 196 flight hours, another within 2151 hours, a very troubling discrepancy of projected reliability. Many public statements claimed that the faults occurred after 52,000 flight hours, but this is not correct. That higher total was taken from the combined totals of all 50 aircraft in service and test flights.

In hindsight, the already weight-optimized Lithium battery should really have been thermally optimized, since temperature is the well-known and well-documented downfall of lithium battery chemistry, especially Lithium-Ion Cobalt Oxide. This battery chemistry is the same one associated with earlier laptop battery failures and fires, and has seen considerable improvement since then. It is worth nothing that most Lithium-ion battery packs picked for electric vehicle packs, where safety is also a key concern, use a different and safer chemistry than the 787 batteries. In previous Toyota hybrids, Nickel-metal hydride packs were used, the Segway transporter uses Lithium Iron Phosphate (LFP) battery chemistry, and the Nissan leaf uses Lithium Manganese Oxide (LMO) chemistry. The Chevy Volt uses Lithium Manganese Spinel chemistry (another name for LMO), and has also licensed Argonne National Lab’s nickel-manganese-cobalt battery technology for possible future use. Lithium-Ion Cobalt Oxide chemistry is conspicuously absent, which is hardly an endorsement.

Also, the known deep-discharge cell-fault issue with Lithium-ion Cobalt Oxide chemistry makes an internal battery start, and especially a repetitive battery start operation a valid concern in an aviation environment. Charging after this severe charge depletion can lead to total battery failure. High temperature operation also leads to rapid stored charge degradation. Actual catastrophic battery failure in only 196 flight hours inevitably has to raise some design concerns regarding the battery chemistry chosen.

The Lithium-ion battery in any form was already a significant weight saving relative to the cold cranking capability of an equivalent Ni-Cad or SLA battery (sealed lead acid), what was really needed was to make the temperamental chemistry less susceptible to thermal failure, and especially from propagating thermal failure from any single failed cell to any others. The unfortunate design choices pushed instead for even less weight, minimizing thermal isolation between cells, thinning internal spacers, and packing the battery cells and battery monitor control circuitry all within one very small container, which then vented into the ambient electrical bay.

This meant that any cell that shorted or overheated could quickly couple its heat to another cell, causing it to also fail. In practice, this rising temperature was eventually high enough to ignite the lithium (which is very difficult to extinguish, as firefighters subsequently learned), leading to serious battery destruction, and volatile venting into the electrical bay. The result was intense heat, corrosive hot vapor and smoke, and of course electrical failure of the battery.

Actual X-ray Battery failure on the ANA 787 main battery on the left, showing dramatic cell deformation compared to the APU battery on the right. Photo courtesy JTSB

Recent NTSB hearings in the latter part of April, 2013 learned that Boeing shot a nail gun into the battery, and failed to start a battery fire, and that Boeing staff had concluded that even an internal battery short could not start a fire as a result. In retrospect they concluded that this was perhaps not a definitive way to validate the battery’s resistance to internal shorts and establishing fire safety.

A further revelation was released in new records on April 23rd at the NTSB hearing, that an actual battery fire caused by an internal short circuit had occurred in 2009 at a subcontractor’s site. This data had not previously been disclosed. Further, NTSB chairman Deborah Hersman noted that cell-to-cell propagation of heat failure (thermal runaway) clearly occurred, a must-not-occur design condition specifically prohibited by the FAA in their agreement to allow the use of Lithium-ion batteries. This design criterion clearly was not met in practice.

Unresolved Other Factors

In addition to general issues with battery chemistry and the realities of engine start operation, several other factors of unknown significance have also surfaced. Once the APU battery was removed as well, they went OFF. This possibly indicates an unintended sneak path of some type between the Main (failed) and APU batteries. This anomaly is still under investigation.

On Jan. 9, 2013, United Airlines, the sole U.S. 787 operator right now, reported wiring problems on one of its six 787s in the same area as the ANA 787 Main battery fire. Details were not disclosed. The JTSB also found a broken battery case grounding strap on the ANA 787, but did not conclude it was contributory to the fire.

One puzzling detail is that other airframes in the flying pool of 50 ships have not had battery fires. Is this because of different starting protocols and methods, different external power sources, different wiring configurations, different flight tines, minor ship differences, or other unknown factors? Also, of great interest to all would be a public disclosure of the visual and operational condition of every battery removed from service when these new retrofit kits are installed. This data may be of critical importance downstream in evaluating the fundamental suitability of the battery chemistry, especially if existing thermal damage is found. Currently no plan to do this has been made public, but it is clearly very useful to do so.

NTSB data specifically ruled out applied battery over-voltage as a fault trigger leading to fire, as did the JTSB. Flight data recorder voltage records (see attached figure) show dramatic internal drops (to 11V and 3V) and surges as if cells are shorting out or shorting to the case, but not an indication of over-voltage. This means internal cell destruction was probably triggered by other causes, which have still not been clearly defined.

Note that the dramatic voltage drop coincides with reaching 32,000 ft altitude, raising the possibility that very low ambient pressure may have caused cell top seal venting or rupturing, and allowed water in to trigger a violent reaction with lithium. Cell vents on both ship’s batteries were damaged, but it is unknown at what point this happened, and whether it is a contributing factor or simply a result of the ensuing fires.

The battery is rated to 32V for normal operation, and only hits 31V in the ANA flight data record.

Getting It Fixed

To satisfy operators and FAA, Boeing re-designed the battery internals (but did not change battery chemistry), increased internal cell spacings and insulation, hardened the internal electronics within the battery housing, had their subcontractors modify the charge control system, enclosed the entire battery in a massive sealed stainless-steel container, and added an outboard titanium exhaust port. Testing showed the new system to be far more robust, and they were unable to re-create any fire scenario within the new pack and its associated system. External venting also solved any problem of secondary system damage within the electrical bay.

The NTSB view of side 4 of the JAL APU battery to show cells C5-C8. Photo courtesy NTSB

There was considerable public discussion after the faults regarding the “self-certification” of many items of the 787 Dreamliner, especially the battery and its associated electrical systems. However, the reality is that FAA oversight of these areas cannot take the place of internal design decisions, and their implications. I believe that W. Edwards Deming was correct when he noted that you cannot test in quality, it must be present from the first to last moment of design and materials selection. The difficulty with the 787 lithium battery system was that testing was used to justify the design, rather than validate it. Testing did not show failures (although one early battery fire and failure did actually occur in battery testing) so therefore the system must be satisfactory. In fact, it merely indicated shortcomings in the test concept itself, something which may be present in the new “fix” as well.

This is the classic case of the considerable hazards of “unknown unknowns.” Those designing a system can only rarely envision all possible fault scenarios and hazards, or correctly weigh them. Simply put, they do not even know what they do not know, and should also be concerned about. They should not be allowed to validate and approve it. Yet, that is what occurred in this case at initial adoption of the technology, with competent oversight, to be sure, and extensive outside review during the correction phase. But as we have seen, it is often better to have initial hostile analysis from outside, non-vested parties when the goal is the best and safest system design. It is a well proven QA strategy in aviation that testing should not be done by the designers, as it is a long way from 1 fault in 10,000,000 flight hours to a fault after only 169 flight hours.

In the corporate universe, considerable importance is placed on meeting schedules (no matter how arbitrarily drawn or without factual foundation), so contrarian positions are not appreciated during technical reviews and problem resolution. Regardless, there is huge value in fostering some of this attitude in every company, as it can prevent everyone from drinking the hemlock when swept up in the awe and majesty of new technology. If both battery platforms had been evaluated, for example, since The Lithium-ion solution was so untried, a simple shift to back-up technology would have been easily possible. This seems like one of those situations where parallel development would have had real value for this airframe, just as multiple engines, interiors and avionics suites are provided on every major airframe.

787 test flights began with Ethiopian airlines on April 27, and ANA (the launch customer of the 787) in Japan on April 28. The world’s 787 fleet should soon be back in the air and fully operational with the updated battery system package.Only further flight testing and customer feedback can prove the efficacy of the repair strategy in this system, but some points should be noted regarding the system design itself in closing. 

Actual battery failure on the ANA 787 main battery. Photo courtesy JTSB

The problematic battery internal chemistry was not changed, presumably because time did not permit a totally new battery concept to be designed, manufactured and tested (and just imagine how well any other lithium-ion battery would have been received by regulatory authorities at that point), so the potential for downstream LiCoO2 chemistry problems remain. Those problems should now be safely contained without threat to the airframe, but they are not actually gone. The weight penalty of the containment system and internal re-design has effectively removed any weight advantage of the Lithium-ion battery system, which of course justifiably raises a few eyebrows as to the fundamental battery choice itself. No doubt the guy that originally proposed a proven ni-cad battery and control system for the ship is feeling pretty vindicated at this point.