Aviation Week & Space Technology
Titan IV Flaws In Software
(Article #19990812014153.aw31.htm.0)
02-Aug-1999 01:41:53 am
ROBERT WALL/WASHINGTON

The need for the U.S. Air Force and contractors to improve Titan IV software quality control and test processes is highlighted in the accident investigation into the Apr. 30 failed launch of a Milstar communications satellite.

The most embarrassing finding for the Titan program is that the $1.23-billion loss could have been avoided if indications of problems had been heeded during the launch process (AW&ST July 26, p. 27). However, of greater long-term significance is the accident investigation board's conclusion that software development and verification processes are flawed.

"The software development process allows single point failures for mission critical data," said the accident investigation board headed by USAF Col. J. Gregory Pavlovich. In this case an incorrectly entered roll rate filter constant--entered as -0.1992476 rather than -1.992476--in the Centaur upper stage's guidance system left the satellite in a useless orbit.

The effects of the incorrect constant manifested themselves about 10 min. after launch, with the start of the Centaur main engine. The Titan upper stage experienced anomalous rolls during the first burn of the Centaur engine. The reaction control system was able to stabilize the Centaur during the first of three coast phases, but burned 85% of its propellant. As a result, there wasn't enough propellant remaining to compensate for control stability that was experienced during the second and third burn phases. The outcome was that Milstar was placed in a 548.7 X 3347.11-mile elliptical Earth orbit, instead of the intended geostationary orbit.

Despite identifying numerous software process deficiencies, investigators "could not identify the single process owner reponsible for understanding, designing, documenting, controlling configuration and ensuring proper execution of the process."

Software process problems included data used in the flight tape not undergoing testing in the Flight Analogous Simulation Test (FAST). The simulation test could have used the actual flight tape data, but software experts who knew that are no longer on the Titan program because of corporate consolidation and program evolution.

Defense Dept. officials with oversight responsibilities "did not have a clear understanding of the process when they developed their surveillance plan." Furthermore, their focus was on controlling areas where previous problems had occurred. Since the roll rate filter constant error hadn't occurred before, the process was deemed low risk.

Moreover, the Titan program office had no full-time organic software support because of downsizing. Even the contractor-provided software support has shrunk by 50% since 1994.

The fact that deficiencies are centered on software is illustrated by the fact that several hardware problems with the Titan IVB and the Centaur were detected before launch and fixed.

Despite all the process shortcomings, there were at least two opportunities to detect the error in the software. About one week before launch, the data tape with the incorrect constant was loaded, and officials raised questions when they observed lower than expected filter rates. But the follow-up process failed to identify the problem because of poor communication between officials.

The second opportunity presented itself on the day of launch when the tower was rolled back. Attitude rates weren't registering the Earth's rotation or impacts of wind. That should have tipped-off launch officials about the problem, but the data wasn't being monitored.


Aviation Week & Space Technology
Panel Links Launch Failures to Systemic Ills
(Article #19990912120532.aw24.htm.0)
13-Sep-1999 12:05:32 pm
WILLIAM B. SCOTT/ COLORADO SPRINGS

Review team faults Lockheed Martin management,engineering, manufacturing and quality control procedures

An independent review team determined that Lockheed Martin's overemphasis on cost-cutting, a loss of experienced personnel, poor quality assurance measures and improper application of "faster, better, cheaper" concepts were largely responsible for launch and spacecraft failures during the last two years.

These and other detailed findings emerged from a four-month investigation by a 16-member "Independent Assessment Team on Mission Success," which focused on the company's Space & Strategic Missiles Sector. Special emphasis was given to the Titan IV program--which only has 11 more scheduled flights--because the heavy-lift booster is considered critical to national security.

The team was led by A. Thomas Young, former president and chief operating officer of Martin Marietta Corp., which merged with Lockheed earlier this decade. Former USAF Vice Chief of Staff Gen. Thomas S. Moorman, Jr. (Ret.), who now is a vice president for Booz-Allen & Hamilton, was the team's vice chairman. About half the team had current or past ties to either Martin Marietta or Lockheed Martin. However, Lockheed Martin executives emphasized that the team's activities were unrestricted and their findings "truly independent." Outsiders included Moorman; Jimmie Hill, former NRO deputy director; Donald L. Cromer, former president of Hughes Space & Communications; Brent Scowcroft, former presidential national security adviser; and Chester L. Whitehair, former vice president of Space Launch Operations at The Aerospace Corp.

The review focused on program management, engineering and manufacturing processes and quality control procedures in the Space and Strategic Missiles Sector, which builds numerous launch vehicles, spacecraft and missile defense systems.

"We spent considerable time on Titan, Atlas, Thaad and Athena. In our review, we . . . found enormous technical strengths throughout the sector. We also found some significant problems," Young said. A comprehensive implementation plan that incorporates all the review's recommended corrective actions is to be developed by late this month. Many of the suggested improvements are already in place.

While detailing systemic problems in Lockheed Martin's launcher and space business areas, the review team emphasized that these ills must be viewed in perspective. "Space and launch are different than most other businesses," Young noted. "It is a 'one-strike-and-you're-out' business. By that I mean, thousands of things can be done correctly, [but] one individual can make one error that causes total mission failure." Consequently, the team dug deep to gain insight into core problems, interviewing approximately 200 people at all levels of the corporation, as well as about two dozen senior government officials.

Their findings and recommendations included:

A requirement for better management of subcontractors andsuppliers. A three-step program is being initiated to emphasize thedefense giant's "mission success" priority, reward suppliers "thatembrace this ethic," and increase oversight to ensure standards aremet, said Albert E. Smith, acting president and COO of LockheedMartin's Space & Strategic Missiles Sector. Smith also is presidentof the Sunnyvale, Calif.-based Missiles and Space Co., but is fillingthe sector's top slot vacated when Thomas A. Corcoran left the company.
Procedures should be reviewed and improved to make them moreresistant to human error.
Workforce quality must be enhanced through better training,mentoring, succession planning and using the skills of retirees tosupport critical events.
Better customer relations should be nurtured. "The team noted someconfusion, both in our workforce and the customer community,about the balancing of our attention to cost objectives and missionsuccess," Smith said. "Clearly, we have a challenge in making surethis is communicated down to all levels."
As examined by Aviation Week & Space Technology's series on "Aerospace in Crisis," the industry's rush to a "faster, better, cheaper" (FBC) way of doing business has resulted in program and job casualties (AW&ST June 21, p. 63). Lockheed Martin's Independent Assessment team agreed, finding that improper implementation of FBC disrupted proven procedures, leading to "mission failures" that cost the company hard-won credibility with customers.

In some cases, adoption of FBC methods were seen as opportunities to discard unwanted design reviews and built-in checks and balances. "When that happened, it typically came back and caused problems," Young said. "The one metric that's easy to measure is 'cheaper.' The 'faster and better' are a bit more difficult to implement."

"There's nothing wrong with 'faster, better, cheaper,' but it's necessary to have a disciplined approach." Without that discipline, a company might "eliminate things that can [then] get you in trouble," Smith added.

The team found that the Theater High-Altitude Air Defense program had significant deficiencies in quality, engineering practices and testing discipline. Major changes are necessary as Thaad moves into the engineering, manufacturing and development phase. For one, better "systems engineering needs to be applied at the front-end of the program to be sure it's done right," Teets said. He noted that retaining key engineering talent, and making sure Thaad is treated as a "weapon system, not just a missile," are imperative. These improvements are in progress, sparked by management changes about a year ago.

Teets refused to fault government policies and actions as contributors to the company's travails. "I believe the problems we've experienced in our launch business are caused by Lockheed Martin, not by the government," he said. Calling the Titan IV/Milstar loss--caused by an improper constant being loaded into a guidance computer's software--"a very rattling experience," Teets faulted the company's internal oversight practices. "That's not the government's fault."

However, other industry experts disagreed. One noted that the "customer's constant pounding on cost-reduction flowed down to the contractors, and they had to respond to stay competitive." Recent cost-sharing initiatives and mandated limitations on corporate profits--even when a technical breakthrough might bolster a contractor's profits--have forced companies to shortcut proven but costly practices, he said. Failures have been the result.

Teets also vigorously rejected a suggestion that company problems may stem from Lockheed Martin becoming too large to manage properly. "No. Absolutely, unequivocally not!" Instead, better local-level leadership is needed "down to the engineering level," with a renewed emphasis on systems engineering, he said.

He implied that no further management changes would be triggered as a result of the assessment team's report. 


Aviation Week & Space Technology
Titan, Delta Failures Force Sweeping Reviews
(Article #19990601002908.aw28-30.htm.0)
10-May-1999 12:29:08 am
CRAIG COVAULT/CAPE CANAVERAL

String of accidents will have broad program impact; Titan/Milstar probe focuses on Lockheed Martin software

A sweeping government/industry reassessment of U.S. space launch capabilities and processes will result from back-to-back failures of the U.S. Air Force/Lockheed Martin Titan IV and Boeing Delta III boosters, delaying major U.S. unmanned launch operations.

Three successive Titan IV mission failures, an Athena failure and two straight mission losses of the large new commercial Delta III, including its latest mishap May 4, mark the worst string of major U.S. launch accidents in 13 years. In addition to rocking U.S. space management, the mishaps could also damage U.S. commercial launch competitiveness.

The Delta III's new hydrogen/oxygen upper stage using a single Pratt & Whitney RL10B-2 engine failed during its burn profile, stranding the Orion 3 communications spacecraft in a useless orbit and dashing Boeing's hopes of returning the Delta III to successful operation (see p. 30).

While losing the $230-million Delta mission is a crisis for Boeing, the situation is even worse for the Air Force and Lockheed Martin, which experienced their third straight Titan IV mission failure on Apr. 30, resulting in the loss of the first USAF Milstar Block 2 communications spacecraft.

The $1.23-billion Titan IV-B/Centaur Milstar mission failure is the single most costly unmanned accident in the 50-year history of Cape Canaveral launch operations. The accident investigation is focusing on Lockheed Martin development, checkout and handling of flight software for the Centaur upper stage. Specific data from the Milstar failure indicate the Centaur upper stage was launched carrying an inaccurate software load from Lockheed Martin that went undetected in the company's software verification process, multiple sources said.

The 10,000-lb. Lockheed Martin Block-2 Milstar built in Sunnyvale, Calif., was to be the single most advanced military communications spacecraft ever placed in operation by the U.S. Air Force. The satellite alone cost $800 million, while the overall Titan IV-B/Centaur launcher cost $433 million. The spacecraft, spanning 51 X 116 ft., is stranded in a useless orbit, its mission officially declared a failure by the Air Force.

USAF has appointed its own standard accident investigation board, to be chaired by Col. Gregory Pavlovich, commander of the 341st Space Wing at Malmstrom AFB, Mont. But the Air Force is also planning a much higher level "broad area review" to assess with the National Reconnaissance Office (NRO) the overall launch crisis.

The White House National Security Council probably will be involved in this or other Defense Dept. assessments of the situation. And senior space managers have also begun to call for congressional oversight hearings on the failures, likely also to cover the Delta III.

Lockheed Martin is conducting its own major internal review. Last week, it appointed A. Thomas Young, former company president and chief operating officer, to chair an independent panel to review Lockheed Martin program management, engineering, manufacturing processes and quality control involved in the Titan situation.

The Milstar flight data show that its Centaur's software, prepared and verified at Lockheed Martin Astronautics near Denver, was corrupted by an incorrect mathematical constant that steered the Centaur off course shortly after ignition for its first burn about 9 min. into the launch. The data also indicate that the software then caused two other abnormal firings of the stage and premature separation of the satellite in a useless, 400 X 2,700-naut.-mi. orbit, instead of a geosynchronous orbit as intended. Some of the anomalous behavior may have been caused by the stage's computers trying to correct for the errors when the system realized it was in trouble, sources said.

How such a critical software error could have slipped through Lockheed Martin's prelaunch software verification process will be a key element of the Air Force accident investigation. The failure has "emotionally devastated" the Lockheed Martin software team and overall Lockheed Martin Astronautics workforce, sources said. That workforce at the Littleton, Colo., plant has already been hurt by the announcement of 900 job cuts and the murder of students at Columbine High School, where the children of many Lockheed Martin employees attend classes.

A major management shakeup at Lockheed Martin Astronautics is also likely from the accident, sources said. It was Lockheed Martin's second failure in three days, following the Apr. 27 loss of the commercial Athena/Ikonos remote-sensing satellite due to a shroud separation problem (AW&ST May 3, p. 45). Investigators believe a mechanical problem caused that failure.

The reviews coming as a result of the failures will have hardware as well as software integration and checkout implications for all U.S. space launch operations, especially for the multibillion-dollar USAF/commercial Evolved Expendable Launch Vehicle (EELV) program managed by both Boeing and Lockheed Martin.

Managers said the latest Titan IV-B and Delta III mission failures could also reinitiate the idea of launching more U.S. military and commercial payloads on the space shuttle.

Last week, Aviation Week & Space Technology aired major USAF and industry concerns about how the previous Titan and Delta failures were crippling U.S. space prowess (AW&ST May 3, p. 31). The subsequent Titan IV-B/Centaur Milstar and Delta III/Orion losses have dramatically exacerbated that situation, although all of the mission failures were caused by seemingly unrelated hardware problems.

The Titan failures "point out the fact that we really do need to go back and review all of that [previous mission data] and build an action plan to say 'this is how we are going to get back on track,'" said USAF Brig. Gen. F. Randall Starbuck, commander of the 45th Space Wing, which manages military launches at Cape Canaveral. "Certainly, when we have three failures in a row of any system or a combination of systems, something is not right," he said.

The 45th Space Wing's 3rd Space Launch Sqdn. (SLS) and hundreds of contractor personnel here managed the checkout and launch of the IV-B/Centaur Milstar vehicle. The software is supplied to the 3rd SLS supposedly verified by Lockheed Martin and ready to fly. The launch team here plays no role in verifying the software, but rather makes sure that it is properly loaded onto the vehicle. No major updates were made to the Milstar software load once it arrived here, officers at the 45th Wing said.

Although the initial Titan Milstar launch was normal, its high altitude phase was bizarre. The 194-ft. Titan IV-B/Centaur with a 76-ft. payload shroud lifted off Pad 40 here on 3.4 million lb. of thrust at 12:30 p.m. on Apr. 30.

The launch had been delayed by 94 min. because the team had fallen slightly behind in preparations after they had to leave the pad temporarily for safety the night before during a severe thunderstorm. Two lightning strikes during the storm landed in the vicinity of the pad, but those bolts fell outside of the large Complex 40 perimeter. Lightning here is routine, and Pad 40 has one of the most sophisticated lighting-protection systems of any launch pad in the world.

Although the Cape was blanketed by low clouds, weather conditions were acceptable for the launch. Data indicate that the Titan IV-B Alliant solid rocket boosters, the Lockheed Martin core vehicle and its Aerojet propulsion worked perfectly through completion of the core's profile 9 min. into the flight.

The Centaur was then to fly a three-burn profile spread over 6 hr. 32 min. to complete the Milstar launch to geosynchronous orbit. The first firing of the two Centaur engines at 9 min. 36 sec. was intended to be a 2-min. firing to place the Centaur/Milstar stack into an initial 97 X 104-naut.-mi. orbit inclined 28 deg. Data show the Centaur's inaccurate software began steering the stage's twin Pratt & Whitney RL10 engines abnormally at that point, sources said. Neither the RL10s nor the Centaur's Honeywell inertial reference unit and computers have been implicated in the failure.

About 30 min. after launch, the team "started picking up indications that the first burn was not exactly what we had expected," said USAF Brig. Gen. Craig R. Cooning, program director for the MilSatCom Joint Program Office at Los Angeles AFB.

Air Force and contractor ground controllers at Schriever AFB, Colo., watched the telemetry that showed the stage behaving erratically, but they were powerless to stop it. The Centaur system is designed so that all functions are fully automatic, with no way to command the stage from the ground to halt abnormal behavior.

After the first burn, the Centaur continued it erratic behavior. It was supposed to ignite for its second burn at 1 hr. 5 min. into the flight northeast of Australia. This was to propel the spacecraft to geosynchronous altitude, where at 6 hr. 22 min. another firing was to halt the spacecraft over the Pacific west of South America. This was to have been followed by spacecraft separation a few minutes later.

Instead of cycling through this more than 6-hr. process, the Centaur's bad software commanded both of the second and third firings within only about 3 hr. of launch, then separated the spacecraft 3 hr. earlier than planned in an orbit 21,000 mi. lower than intended.

Air Force Space and Missile System Center (SMC) and contractor personnel from Los Angeles, located at Schriever for the launch, immediately began sending commands to the spacecraft from Schriever's SOC-42 Space Operations Center, according to Col. Mike Kelly, deputy commander of the 50th Operations Group at the base.

The objective was to save the spacecraft to preserve whatever options might remain. The new Block 2, which took more than three years to build, was to provide substantially more capability than the previous two Milstars launched in 1994 and 1995, including a new ability to transmit data at up to 1.5 megabits per sec. It was to have been parked over the Galapagos Islands, but not become operational until September.

The controllers succeeded in stabilizing the Milstar's attitude and deployed its 116-ft. solar arrays and truss-like payload wings spanning 51 ft. Although the spacecraft is electrically healthy, the mission is a failure from a defense communications standpoint because its antennas are not designed to slew to hold a ground footprint and send data from such a low altitude. Engineers are studying whether any engineering or communications tests can be done with the vehicle.

The Milstar failure is the third since last Aug. 12, when a Titan IV-A/Centaur carrying an NRO Mercury signals intelligence spacecraft exploded 40 sec. into flight. That $1-billion failure, the previous worst unmanned accident here, was blamed on a wiring fault in the core vehicle's second stage.

The second mission loss occurred Apr. 9 when the solid-propellant Boeing Inertial Upper Stage for another Titan IV-B failed to place a Defense Support Program missile warning satellite into geosynchronous orbit. Last week, the Air Force confirmed earlier Aviation Week reports that IUS first- and second-stage separation problems and related second-stage nozzle damage are the focus of that investigation (AW&ST May 3, p. 46).

Data from the $682-million DSP mission failure show that the first-stage IUS separation was not complete, there were second-stage nozzle extension anomalies and the vehicle tumbled after the second stage fired.

©The McGraw-Hill Companies Inc.


See also the USAF investigation report (Microsoft Word format).