http://www.me.gatech.edu/me/publicat/AugTranscript.htm

Excerpt from: "Yes, But Will It Work in Theory?"
Norman R. Augustine
1996 Woodruff Distinguished Lecture Transcript

No Change Is a Small Change

The company I work for builds the Titan space launch vehicle. Some years ago, we adapted a Titan to put commercial payloads into orbit. This model was designed to carry two payloads - two spacecraft - within its large payload fairing, and hence was designed with two ports through which the wiring and umbilical cords were hooked up. Now, in this particular instance, there was going to be only one payload, albeit a very large one. The engineers had a choice of wiring the payload either through the front port or through the aft port.

As it happened, the engineer on the job chose the drawings to do the wiring through the front port, and the software was designed according to that choice. That engineer was then shifted to another project. About a year and a half later, another engineer was assigned to the project and observed, "That's kind of crazy. Why should we run wires 25 feet up the side of the launch vehicle? Why not plug into the aft port?" So he redlined the drawings, changed the configuration to "improve" it, and showed that the wiring should be plugged into the aft port. The only problem was that somewhere along the line, we had designed in an escape vent in our configuration control system, and the people who wrote the software wrote it to expect the input from the front port.

The launch day came, and the Titan flew beautifully. The giant vehicle separated at precisely the right moment, the second stage fired perfectly, and the vehicle attained the orbit just where it was supposed to be. Everything was fine. Then we sent the signal to separate the payload, and nothing happened. We couldn't separate it; nothing could be done. You might ask, "Why didn't the project team catch the problem during the systems testing that preceded the launch?" The reason for that presents another lesson of how things go wrong: The system tests used generic software instead of the actual flight software. The generic software, as we discovered later to our great regret, accepted input from either port, so the separation "worked" perfectly in theory since it wasn't particular with regard to which port gave the signal. This led to a $300 million lawsuit and kept our legal department happy for two years.