The
need for the U.S. Air Force and contractors to improve Titan IV software
quality control and test processes is highlighted in the accident investigation
into the Apr. 30 failed launch of a Milstar communications satellite.
The
most embarrassing finding for the Titan program is that the $1.23-billion
loss could have been avoided if indications of problems had been heeded
during the launch process (AW&ST July 26, p. 27). However, of greater
long-term significance is the accident investigation board's conclusion
that software development and verification processes are flawed.
"The
software development process allows single point failures for mission critical
data," said the accident investigation board headed by USAF Col. J. Gregory
Pavlovich. In this case an incorrectly entered roll rate filter constant--entered
as -0.1992476 rather than -1.992476--in the Centaur upper stage's guidance
system left the satellite in a useless orbit.
The
effects of the incorrect constant manifested themselves about 10 min. after
launch, with the start of the Centaur main engine. The Titan upper stage
experienced anomalous rolls during the first burn of the Centaur engine.
The reaction control system was able to stabilize the Centaur during the
first of three coast phases, but burned 85% of its propellant. As a result,
there wasn't enough propellant remaining to compensate for control stability
that was experienced during the second and third burn phases. The outcome
was that Milstar was placed in a 548.7 X 3347.11-mile elliptical Earth
orbit, instead of the intended geostationary orbit.
Despite
identifying numerous software process deficiencies, investigators "could
not identify the single process owner reponsible for understanding, designing,
documenting, controlling configuration and ensuring proper execution of
the process."
Software
process problems included data used in the flight tape not undergoing testing
in the Flight Analogous Simulation Test (FAST). The simulation test could
have used the actual flight tape data, but software experts who knew that
are no longer on the Titan program because of corporate consolidation and
program evolution.
Defense
Dept. officials with oversight responsibilities "did not have a clear understanding
of the process when they developed their surveillance plan." Furthermore,
their focus was on controlling areas where previous problems had occurred.
Since the roll rate filter constant error hadn't occurred before, the process
was deemed low risk.
Moreover,
the Titan program office had no full-time organic software support because
of downsizing. Even the contractor-provided software support has shrunk
by 50% since 1994.
The
fact that deficiencies are centered on software is illustrated by the fact
that several hardware problems with the Titan IVB and the Centaur were
detected before launch and fixed.
Despite
all the process shortcomings, there were at least two opportunities to
detect the error in the software. About one week before launch, the data
tape with the incorrect constant was loaded, and officials raised questions
when they observed lower than expected filter rates. But the follow-up
process failed to identify the problem because of poor communication between
officials.
The
second opportunity presented itself on the day of launch when the tower
was rolled back. Attitude rates weren't registering the Earth's rotation
or impacts of wind. That should have tipped-off launch officials about
the problem, but the data wasn't being monitored.
Aviation Week & Space Technology
Panel Links Launch Failures to Systemic Ills
(Article #19990912120532.aw24.htm.0)
13-Sep-1999 12:05:32 pm
WILLIAM B. SCOTT/ COLORADO SPRINGS
Review
team faults Lockheed Martin management,engineering, manufacturing and quality
control procedures
An
independent review team determined that Lockheed Martin's overemphasis
on cost-cutting, a loss of experienced personnel, poor quality assurance
measures and improper application of "faster, better, cheaper" concepts
were largely responsible for launch and spacecraft failures during the
last two years.
These
and other detailed findings emerged from a four-month investigation by
a 16-member "Independent Assessment Team on Mission Success," which focused
on the company's Space & Strategic Missiles Sector. Special emphasis
was given to the Titan IV program--which only has 11 more scheduled flights--because
the heavy-lift booster is considered critical to national security.
The
team was led by A. Thomas Young, former president and chief operating officer
of Martin Marietta Corp., which merged with Lockheed earlier this decade.
Former USAF Vice Chief of Staff Gen. Thomas S. Moorman, Jr. (Ret.), who
now is a vice president for Booz-Allen & Hamilton, was the team's vice
chairman. About half the team had current or past ties to either Martin
Marietta or Lockheed Martin. However, Lockheed Martin executives emphasized
that the team's activities were unrestricted and their findings "truly
independent." Outsiders included Moorman; Jimmie Hill, former NRO deputy
director; Donald L. Cromer, former president of Hughes Space & Communications;
Brent Scowcroft, former presidential national security adviser; and Chester
L. Whitehair, former vice president of Space Launch Operations at The Aerospace
Corp.
The
review focused on program management, engineering and manufacturing processes
and quality control procedures in the Space and Strategic Missiles Sector,
which builds numerous launch vehicles, spacecraft and missile defense systems.
"We
spent considerable time on Titan, Atlas, Thaad and Athena. In our review,
we . . . found enormous technical strengths throughout the sector. We also
found some significant problems," Young said. A comprehensive implementation
plan that incorporates all the review's recommended corrective actions
is to be developed by late this month. Many of the suggested improvements
are already in place.
While
detailing systemic problems in Lockheed Martin's launcher and space business
areas, the review team emphasized that these ills must be viewed in perspective.
"Space and launch are different than most other businesses," Young noted.
"It is a 'one-strike-and-you're-out' business. By that I mean, thousands
of things can be done correctly, [but] one individual can make one error
that causes total mission failure." Consequently, the team dug deep to
gain insight into core problems, interviewing approximately 200 people
at all levels of the corporation, as well as about two dozen senior government
officials.
Their
findings and recommendations included:
In
some cases, adoption of FBC methods were seen as opportunities to discard
unwanted design reviews and built-in checks and balances. "When that happened,
it typically came back and caused problems," Young said. "The one metric
that's easy to measure is 'cheaper.' The 'faster and better' are a bit
more difficult to implement."
"There's nothing wrong with 'faster, better, cheaper,' but it's necessary to have a disciplined approach." Without that discipline, a company might "eliminate things that can [then] get you in trouble," Smith added.
The
team found that the Theater High-Altitude Air Defense program had significant
deficiencies in quality, engineering practices and testing discipline.
Major changes are necessary as Thaad moves into the engineering, manufacturing
and development phase. For one, better "systems engineering needs to be
applied at the front-end of the program to be sure it's done right," Teets
said. He noted that retaining key engineering talent, and making sure Thaad
is treated as a "weapon system, not just a missile," are imperative. These
improvements are in progress, sparked by management changes about a year
ago.
Teets
refused to fault government policies and actions as contributors to the
company's travails. "I believe the problems we've experienced in our launch
business are caused by Lockheed Martin, not by the government," he said.
Calling the Titan IV/Milstar loss--caused by an improper constant being
loaded into a guidance computer's software--"a very rattling experience,"
Teets faulted the company's internal oversight practices. "That's not the
government's fault."
However,
other industry experts disagreed. One noted that the "customer's constant
pounding on cost-reduction flowed down to the contractors, and they had
to respond to stay competitive." Recent cost-sharing initiatives and mandated
limitations on corporate profits--even when a technical breakthrough might
bolster a contractor's profits--have forced companies to shortcut proven
but costly practices, he said. Failures have been the result.
Teets also vigorously rejected a suggestion that company problems may stem from Lockheed Martin becoming too large to manage properly. "No. Absolutely, unequivocally not!" Instead, better local-level leadership is needed "down to the engineering level," with a renewed emphasis on systems engineering, he said.
He
implied that no further management changes would be triggered as a result
of the assessment team's report.
Aviation Week & Space Technology
Titan, Delta Failures Force Sweeping Reviews
(Article #19990601002908.aw28-30.htm.0)
10-May-1999 12:29:08 am
CRAIG COVAULT/CAPE CANAVERAL
String
of accidents will have broad program impact; Titan/Milstar probe focuses
on Lockheed Martin software
A
sweeping government/industry reassessment of U.S. space launch capabilities
and processes will result from back-to-back failures of the U.S. Air Force/Lockheed
Martin Titan IV and Boeing Delta III boosters, delaying major U.S. unmanned
launch operations.
Three successive Titan IV mission failures, an Athena failure and two straight mission losses of the large new commercial Delta III, including its latest mishap May 4, mark the worst string of major U.S. launch accidents in 13 years. In addition to rocking U.S. space management, the mishaps could also damage U.S. commercial launch competitiveness.
The Delta III's new hydrogen/oxygen upper stage using a single Pratt & Whitney RL10B-2 engine failed during its burn profile, stranding the Orion 3 communications spacecraft in a useless orbit and dashing Boeing's hopes of returning the Delta III to successful operation (see p. 30).
While losing the $230-million Delta mission is a crisis for Boeing, the situation is even worse for the Air Force and Lockheed Martin, which experienced their third straight Titan IV mission failure on Apr. 30, resulting in the loss of the first USAF Milstar Block 2 communications spacecraft.
The $1.23-billion Titan IV-B/Centaur Milstar mission failure is the single most costly unmanned accident in the 50-year history of Cape Canaveral launch operations. The accident investigation is focusing on Lockheed Martin development, checkout and handling of flight software for the Centaur upper stage. Specific data from the Milstar failure indicate the Centaur upper stage was launched carrying an inaccurate software load from Lockheed Martin that went undetected in the company's software verification process, multiple sources said.
The 10,000-lb. Lockheed Martin Block-2 Milstar built in Sunnyvale, Calif., was to be the single most advanced military communications spacecraft ever placed in operation by the U.S. Air Force. The satellite alone cost $800 million, while the overall Titan IV-B/Centaur launcher cost $433 million. The spacecraft, spanning 51 X 116 ft., is stranded in a useless orbit, its mission officially declared a failure by the Air Force.
USAF has appointed its own standard accident investigation board, to be chaired by Col. Gregory Pavlovich, commander of the 341st Space Wing at Malmstrom AFB, Mont. But the Air Force is also planning a much higher level "broad area review" to assess with the National Reconnaissance Office (NRO) the overall launch crisis.
The White House National Security Council probably will be involved in this or other Defense Dept. assessments of the situation. And senior space managers have also begun to call for congressional oversight hearings on the failures, likely also to cover the Delta III.
Lockheed Martin is conducting its own major internal review. Last week, it appointed A. Thomas Young, former company president and chief operating officer, to chair an independent panel to review Lockheed Martin program management, engineering, manufacturing processes and quality control involved in the Titan situation.
The Milstar flight data show that its Centaur's software, prepared and verified at Lockheed Martin Astronautics near Denver, was corrupted by an incorrect mathematical constant that steered the Centaur off course shortly after ignition for its first burn about 9 min. into the launch. The data also indicate that the software then caused two other abnormal firings of the stage and premature separation of the satellite in a useless, 400 X 2,700-naut.-mi. orbit, instead of a geosynchronous orbit as intended. Some of the anomalous behavior may have been caused by the stage's computers trying to correct for the errors when the system realized it was in trouble, sources said.
How such a critical software error could have slipped through Lockheed Martin's prelaunch software verification process will be a key element of the Air Force accident investigation. The failure has "emotionally devastated" the Lockheed Martin software team and overall Lockheed Martin Astronautics workforce, sources said. That workforce at the Littleton, Colo., plant has already been hurt by the announcement of 900 job cuts and the murder of students at Columbine High School, where the children of many Lockheed Martin employees attend classes.
A major management shakeup at Lockheed Martin Astronautics is also likely from the accident, sources said. It was Lockheed Martin's second failure in three days, following the Apr. 27 loss of the commercial Athena/Ikonos remote-sensing satellite due to a shroud separation problem (AW&ST May 3, p. 45). Investigators believe a mechanical problem caused that failure.
The reviews coming as a result of the failures will have hardware as well as software integration and checkout implications for all U.S. space launch operations, especially for the multibillion-dollar USAF/commercial Evolved Expendable Launch Vehicle (EELV) program managed by both Boeing and Lockheed Martin.
Managers said the latest Titan IV-B and Delta III mission failures could also reinitiate the idea of launching more U.S. military and commercial payloads on the space shuttle.
Last week, Aviation Week & Space Technology aired major USAF and industry concerns about how the previous Titan and Delta failures were crippling U.S. space prowess (AW&ST May 3, p. 31). The subsequent Titan IV-B/Centaur Milstar and Delta III/Orion losses have dramatically exacerbated that situation, although all of the mission failures were caused by seemingly unrelated hardware problems.
The Titan failures "point out the fact that we really do need to go back and review all of that [previous mission data] and build an action plan to say 'this is how we are going to get back on track,'" said USAF Brig. Gen. F. Randall Starbuck, commander of the 45th Space Wing, which manages military launches at Cape Canaveral. "Certainly, when we have three failures in a row of any system or a combination of systems, something is not right," he said.
The 45th Space Wing's 3rd Space Launch Sqdn. (SLS) and hundreds of contractor personnel here managed the checkout and launch of the IV-B/Centaur Milstar vehicle. The software is supplied to the 3rd SLS supposedly verified by Lockheed Martin and ready to fly. The launch team here plays no role in verifying the software, but rather makes sure that it is properly loaded onto the vehicle. No major updates were made to the Milstar software load once it arrived here, officers at the 45th Wing said.
Although the initial Titan Milstar launch was normal, its high altitude phase was bizarre. The 194-ft. Titan IV-B/Centaur with a 76-ft. payload shroud lifted off Pad 40 here on 3.4 million lb. of thrust at 12:30 p.m. on Apr. 30.
The launch had been delayed by 94 min. because the team had fallen slightly behind in preparations after they had to leave the pad temporarily for safety the night before during a severe thunderstorm. Two lightning strikes during the storm landed in the vicinity of the pad, but those bolts fell outside of the large Complex 40 perimeter. Lightning here is routine, and Pad 40 has one of the most sophisticated lighting-protection systems of any launch pad in the world.
Although the Cape was blanketed by low clouds, weather conditions were acceptable for the launch. Data indicate that the Titan IV-B Alliant solid rocket boosters, the Lockheed Martin core vehicle and its Aerojet propulsion worked perfectly through completion of the core's profile 9 min. into the flight.
The Centaur was then to fly a three-burn profile spread over 6 hr. 32 min. to complete the Milstar launch to geosynchronous orbit. The first firing of the two Centaur engines at 9 min. 36 sec. was intended to be a 2-min. firing to place the Centaur/Milstar stack into an initial 97 X 104-naut.-mi. orbit inclined 28 deg. Data show the Centaur's inaccurate software began steering the stage's twin Pratt & Whitney RL10 engines abnormally at that point, sources said. Neither the RL10s nor the Centaur's Honeywell inertial reference unit and computers have been implicated in the failure.
About 30 min. after launch, the team "started picking up indications that the first burn was not exactly what we had expected," said USAF Brig. Gen. Craig R. Cooning, program director for the MilSatCom Joint Program Office at Los Angeles AFB.
Air Force and contractor ground controllers at Schriever AFB, Colo., watched the telemetry that showed the stage behaving erratically, but they were powerless to stop it. The Centaur system is designed so that all functions are fully automatic, with no way to command the stage from the ground to halt abnormal behavior.
After the first burn, the Centaur continued it erratic behavior. It was supposed to ignite for its second burn at 1 hr. 5 min. into the flight northeast of Australia. This was to propel the spacecraft to geosynchronous altitude, where at 6 hr. 22 min. another firing was to halt the spacecraft over the Pacific west of South America. This was to have been followed by spacecraft separation a few minutes later.
Instead of cycling through this more than 6-hr. process, the Centaur's bad software commanded both of the second and third firings within only about 3 hr. of launch, then separated the spacecraft 3 hr. earlier than planned in an orbit 21,000 mi. lower than intended.
Air Force Space and Missile System Center (SMC) and contractor personnel from Los Angeles, located at Schriever for the launch, immediately began sending commands to the spacecraft from Schriever's SOC-42 Space Operations Center, according to Col. Mike Kelly, deputy commander of the 50th Operations Group at the base.
The objective was to save the spacecraft to preserve whatever options might remain. The new Block 2, which took more than three years to build, was to provide substantially more capability than the previous two Milstars launched in 1994 and 1995, including a new ability to transmit data at up to 1.5 megabits per sec. It was to have been parked over the Galapagos Islands, but not become operational until September.
The controllers succeeded in stabilizing the Milstar's attitude and deployed its 116-ft. solar arrays and truss-like payload wings spanning 51 ft. Although the spacecraft is electrically healthy, the mission is a failure from a defense communications standpoint because its antennas are not designed to slew to hold a ground footprint and send data from such a low altitude. Engineers are studying whether any engineering or communications tests can be done with the vehicle.
The Milstar failure is the third since last Aug. 12, when a Titan IV-A/Centaur carrying an NRO Mercury signals intelligence spacecraft exploded 40 sec. into flight. That $1-billion failure, the previous worst unmanned accident here, was blamed on a wiring fault in the core vehicle's second stage.
The second mission loss occurred Apr. 9 when the solid-propellant Boeing Inertial Upper Stage for another Titan IV-B failed to place a Defense Support Program missile warning satellite into geosynchronous orbit. Last week, the Air Force confirmed earlier Aviation Week reports that IUS first- and second-stage separation problems and related second-stage nozzle damage are the focus of that investigation (AW&ST May 3, p. 46).
Data from the $682-million DSP mission failure show that the first-stage IUS separation was not complete, there were second-stage nozzle extension anomalies and the vehicle tumbled after the second stage fired.
©The McGraw-Hill Companies Inc.
See also the USAF investigation report (Microsoft Word format).