VADS was not a perfect laboratory for comparing languages: each feature was (normally) done in only one language. What if C was used for the hardest problems and Ada was used for easier problems?

Unfortunately difficulty of project is not easily measured independent of language. Most experts describe real-time code as the most expensive to build and integrate; Ada was used for most of the real time needs of VADS. Experts also agree that sheer size of modules will produce bugs with non-linear growth; C was used for the biggest, oldest sections of VADS. Our experience indicates that dependence on external, changing entities such as OSs, object formats, and optimization requests are one of the most troublesome burdens for development; all of VADS has these sorts of dependencies, though some more than others.

The C language was used for older parts of VADS, and those features directly associated with the older parts of VADS, including the core VADS front end(>200k lines), the core debugger(>130k lines), and the core code generator(>25k lines). C was used for machine-dependent debugger areas (>15k lines each) and code generator areas(>20k lines each in addition to the cores. The debugger includes some difficult areas with real-time programming needs. The C code also shows some big directories that have a lot of duplication and therefore are easier to produce, in the code generator and initialization areas. We also see some directories strongly associated with external sources, for example X, Motif, editors, and Curses.

Ada was generally used for most areas developed after the basic compiler units, and without a strong C legacy from external code. The top 25 largest Ada directories show emphasis on interfaces both to external libraries such as X and from internal data bases such as DIANA. The most difficult code in general is the real time support code, most of which is in smaller units than these big directories but that shows up in the basic runtimes, the Fault Tolerance extensions, the networking support, the target debug support, and the file support systems. Some tool support is present in the form of the optimizer and the cross linker.

The VADS product line involves more than two thousand directories. By measuring the time spent by developers in these directories, we can calculate a rough measure of their "expense", and examine trends in these more expensive directories to see if they are consistent with the overall trends. The top ten most expensive C directories had these bug fix rates:

  Fixes/Feature    Project Directory     Source Lines         Developers     

       .98             vms_tools             7743                 25         

       .93             amd29k_cg             14248                14         

       .79                 fe               209346                52         

       .61                 d                132492                53         

       .44                 cg                23975                42         

       .63               optim3              13713                14         

       .58              m68k_cg              18527                23         

       .56               unix_d              20307                45         

       .54                mw_d               40469                9          

       .17               xwin_d              23292                8          

Of the top 25 most expensive directories, 80% were C directories. We have to look farther to find ten expensive Ada directories:

  Fixes/Feature    Project Directory     Source Lines         Developers     

       .71             lib_tools             13689                43         

       .66             posix_ada             17759                6          

       .64                sup                18083                43         

       .47               xlink               16052                15         

       .38               optim4              42199                17         

       .38              test_sup             9313                 30         

       .21               new_ts              20438                27         

       .18                inet               18087                4          

       .14              i860_cg              12342                10         

       .12              new_krn              25113                18         

                                                                             

Where there are many contributors in every project, we can assume again that our numbers will be statistically unbiased by individual styles. There is no correlation between the number of contributors to a project and its bug fix rate, indicating that the development environment supported parallel developments.

Average fixes/feature in C for       .62                                   
"hard" projects:                                                           

Average fixes/feature in Ada for     .39                                   
"hard" projects:                                                           

It happens that no project was begun in both C and Ada independently. However, several projects were recoded in Ada after beginning in C. There were two motivations for going to Ada at those early times: some were motivated simply because we wanted to be using our own tools; others were moved to Ada because we need to get code onto a cross target where no C compilers existed. One example is the optimizer:

  Fixes/Feature    Project Directory     Source Lines         Developers     

       .63               optim3              13713                14         

       .38            optim4(Ada)            42199                17         

The Ada-based optimizer project was basically a "start-from-scratch", using an entirely different optimizer data structure and algorithms, and with completely different personnel. The "optim4" optimizer now does more optimizations with greater reliability than did the C-based optimizer.

The optimizer project, being a self-contained user-visible tool of critical importance to at least some users, provides another metric: field-reported bugs:

   Project        Critical         Serious          Minor         Years in    
                                                                  Service     

 optim4(Ada)          3               7               0            4 yr.      

    optim3           13              16               2            4 yr.      

The optimizer shows consistency with a general theme, reported later: we saw more bugs escape our testing and getting through to users from C-based code.

Another example coded from C to Ada was the library tools project for VMS. The tools project provide Ada library tools (e.g., create, remove, clean) for VMS.

Fixes/Feature      Project Directory  Source Lines        Developers         

.98                vms_tools          7743                25                 

.22                vms_ada_tool(Ada)  8900                12                 

In this project there was substantial value carried from the old design, so that the Ada version was able to get a better start.

The runtime was also rewritten in Ada from C but no definitive data survives for the original C effort since development on it occurred mainly prior to 1987 when the source code control records became more accurate.

Of Verdix hires, 75% had done substantial C programming before joining the VADS project. Less than 25% had done substantial programming in Ada before their hiring. No attempt was made to target Ada experts as hires, though because the business of selling Ada brought the company contact with and recognition from the Ada industry, some experienced hires were made from Ada backgrounds.

Training was very much "on the job" and "self imposed". Most developers learned by following the code styles that they saw already established in the code they worked from. While few formal code reviews ever occurred, VADS developers expected a constant peer review as other developers wandered in and out of their code. Those with bugs could expect more "help" than those executing flawlessly. Thus, the largest C project was never more than four developers yet records show that nearly every person ever involved in VADS has changed that front end project.

In the following graphs, we compare fix rates with respect to experience, salary and rating. If Ada personnel are substantially different than C personnel we might see patterns in these comparisons.

The above graph plots two spots for each developer: each solid spot represents the average number of fixes per feature for one of the 60 contributing developers when coding in C, while each open spot represents the same average for Ada contributions. There are one C and one Ada spot for each developer; The several spots at (0,0) represent those few people who either never did Ada or who never did C.

We see that C and Ada fix rates are not dissimilar among those with little experience as measured by number of updates, but that as experience increases that Ada fix rates decline significantly. This training effect will be explored in the next section. We also see that there are more experienced people are generally working with C, as is expected since the product line started with C.

In general, developers each do better in Ada than in C, regardless of their level of experience and salary. Only three developers did significant updates for both C and Ada and had lower fix rates for C, and that by only narrow margins. Again we see that more junior people tend to be more in Ada. We also see a small correlation with increasing salary for Ada programming, in that bug rates decline slightly with salary. This effect is seen more clearly in a later graph, where experience as measured in Ada feature updates is compared with fix rates.

Salary does not necessarily reflect skill or general programming effectiveness. We can compare employee rating with bug fix rates and get a measure of where the most skilled people are working:

The following graph shows the average contributions of developers in each successive six months after joining the VADS effort. In this graph, the X-axis values represent half-year periods with the project. For example, we see that on the average for all contributors who stayed with the project at least four years, these contributors averaged about .48 fixes per C feature and about .23 features per Ada feature in the period between their 3.5 year and 4th year with the project.

This graph indicates that Ada fixes/feature stay relatively steady until about the 7th year with the project when the number of fixes rises dramatically and erratically. C fix rates start higher but decline through the mid-years, then rise and finally become erratic. The rise in fix rates for those in their seventh or more years with the project is explainable by the gradual shift from developer to manager for those who had been longest with the project; managers typically do much fewer features and spend time helping track down bugs and fixes. The next graph confirms this observation

Fix rates for Ada start low and stay low until we get out amongst the longtime employees. Of particular interest is that there does not seem to be a "startup cost" with using Ada compared to C. The longtime employees tend to spend more time in C.

The above data does not necessarily tell us how well languages do for expert users, however. The above charts can tell us whet our average fix rates us but it does not really capture language-specific knowledge. It could be that people were make feature additions or changes in areas where they knew little. We can look at language-specific experience instead of gross project experience.

The following graph presents data on learning curves as fix rates for the two languages based on cumulative number of features added for each language. For example, the people who have, in their total contribution on the project, added between 0 and 500 features in C had an average fix rate of .28 fixes/feature.

The fixes per feature rates show a decline in Ada but actually rises in C. Note that values after 2500 features are statistically unreliable as only a few (supercoder) individuals are represented. Even so, we see a distinct rise in bugs for C and a decline in Ada. In each case, more experienced people are likely moving into more difficult contribution areas.

The VADS bug rates, especially for released products, have been impressively low, whether for C or Ada. Coding standards for C have been aimed at avoiding the known problems with the C language. A partial list is:

• · dangling "else"s
• · use of "=" for "==", especially in conditional statements
• · use of "/=" for "!="
• · overuse of macros
• · use of "cute programming" that sacrifices comprehension for brevity
• · use of integer for pointer

Since C compilers have their share of hard-to-find bugs, and incompatibilities of interpretation from on vendor's C compiler to another, we adopted the general rule that we would avoid buggy areas of compilers as well as of the language itself;

• · avoid using the C compiler's higher optimization levels
• · avoid using complex data structuring of nested unions and structures

The most important internal standards have to do with intentional checks and debugability. VADS was built with internal assertions to check both its C and Ada code. We called this "executable documentation". Asserts in C are designed to make sure that data structures are consistent with each algorithm's expectations both on entry to and exit from major functional areas. Assertions might be simple ("pointer != null") or as complex as running two different algorithms and comparing the results.

Some assertions are expensive enough that they cannot be left on at all times, but most are left on even for the product releases in the field; on most compiles the VADS front end expends more than 10% of its time on internal self checks. As a result, most VADS problems are reported as "assertion errors". While assertion errors occasionally represent false positives, they greatly enhance and early fault detection and debugging.

To further enhance assertion check effectiveness, VADS has internal stress automation: some conditions are very rare and difficult to produce in user code in the presence of shifting optimizations or usage, so stress automation is constructed to force the conditions to cover each case. An example is for code generation when registers are all busy, a difficult condition to produce on RISC machines with hundreds of registers. Stress automation is more effective when working with our TestMate code coverage tools but these were not available during the course of this data.

The VADS debugging tools, including the debugger, its new windowed version, the trace tool, the profiling tool, and the code coverage and test frameworks tools are all designed and used equally well with C or Ada. However, with very large programs no tool can be efficient without customization. VADS developers were expected to generate debugging routines to work in conjunction with the debugger to help isolate problems in complex data structures.

There are more assertions in C code than in Ada code. Ada has stricter typing and runtime checks, so fewer assertions were needed.

The cost of adding features and debugging them is a major part of the engineering costs for products. The engineering costs also include building, testing and releasing products. Their costs would correlate with bug rates since building and testing products has high automation that is broken by bugs. Costs of build/test are generally calculated to be about 35% of the overall development expense of VADS, compared with about 50% spent on new features and enhancements.

Customer Support for VADS represents another cost both inside development as bugs are reported and fixed, and outside development as salespeople, field support and customer service representatives try to assist the customers. Many problems are not the "fault" of the tools themselves, so only a part of the Customer Service cost can be attributed to the tools. However, tool faults are a significant cost both in support and in sales.

Our Customer Support team's records on customer questions and complaints show the following:

These complaints arose from all aspects of the product line, but many could be attributed to specific parts of our implementation. The following graph summarizes the source of bugs by the implementation language used:

This graph shows that customers were far more unhappy with features implemented in C than they were in Ada. The following graph shows the history of Customer Support complaints attributable to C-based and Ada-Based potions of released product.

This table shows a very encouraging overall trend to fewer customer complaints. We hope that trend will continue. That trend shows more clearly in C-based code than Ada-based. Unfortunately there may be some complicating circumstances. For one, in the last year the merger of the two companies has significantly dislocated both our customer support organization and likely our customers; Thus the 1993.5-1994 numbers for DRs are not likely accurate. For another, since VADS is supported on many hosts and targets, and since some features are available only on a limited basis, the usage of new features and thus their reported bugs tends to lag product introduction. Finally, field reports are dependent on circumstances such as poor corporate performance in 1987, and major new releases in 1991.

In comparing C-based DRs with Ada-based DRs, one could ask whether there was not simply many more lines of C code in use by customers. Prior to 1991 there were about 200k fewer Ada SLOC in customer hands. By 1991, the two code bases had reached parity at roughly 1,500,000 SLOC each.

This table also shows an significant majority of reported user problems as arising from C-based code. Bugs are, unfortunately, not reported uniformly for all aspects of the product except by a few very thorough customers or partners. Therefore one could ask whether C was being used for areas that were more critical to customers and so bugs would more likely cause complaint. We can judge criticality by comparing the ratio of critical complaints against other complaints, on the grounds that users will elevate the importance of a problem based on its impact to them.

On this basis, criticality for various components was judged:

Runtime System             Ada      0.75    

Optimizer                   C       0.72    

Optimizer                  Ada      0.42    

Code Generator              C       0.41    

Cross Linker               Ada      0.28    

Front End                   C       0.27    

Library Services           Ada      0.18    

General problems            ?       0.08    

Prelinker                   C       0.06    

Autobuilder                 C       0.03    

Debugger                    C       0.00    

Support Tools              Ada      0.00    

User Documentation          ?       0.00    

We have related a preliminary finding that Ada performed twice as well as C. Subsequent analysis showed that on a feature by feature basis that Ada was not quite that effective for pure code primarily because redundant code in specifications tended to raise lines of code, but that when effects of makefiles and of external costs were factored back in that Ada costs would be on the order of twice as effective, or half the cost, of our carefully-crafted C. Why?

We have done some analysis of our general development methodology and of our errors. The simple answer is that Ada encourages its users to spend more time in writing to describe their code. That extra effort communicates information to the tools and to all future readers of that code. Future readers then derive benefits, including:

Better Locality of Error. The Ada user tends to be able to assume a lot more about where a problem is arising. Often the bugs are indicated directly by the compiler. When code compiles, it has a higher probability of successfully executing. If it fails, the developer will have to look in fewer places to find the cause. We call this effect "error locality": an error is local in time if it is discovered very soon after it is created; an error is local in space if it is identified very close (at) the site where the error actually resides.

Better Tool Support. The extra information provided to Ada tools by its users are both a burden and a blessing. For the first several years, the extra information was a burden to Ada tool vendors because they had to do considerably more work to absorb and check the extra information. Now that same information is a blessing that allows the tools to do more for its users. The effect is most pronounced for tasking, where the Ada tools can create parallel, distributed multiprocessing versions of ordinary-looking programs without the users having to do much more than they did for a single processor. Another big win is in machine code, where users can get free access to the underlying hardware but not have to give up the semantic checks and supports of the high level language.

Reduced Effective Complexity. Although Ada is a more complex language than C, the ultimate complexity is a function of the job being done; accomplishing useful complexity within the language may reduce the overall complexity of the developer's job.

The costs of C should be more pronounced in most other organizations. The VADS team had several advantages that should have made language choice less important:

4) Many teams do not take our aggressive approaches in trying to avoid known C problems and detect problems earlier with thorough testing. We also had advantages of existing test suites and of many users.

Some may look at this study and conclude that C++ will tame C's problems.

Our early experience does not support that conclusion.

Ada94 is the new version of Ada. It includes OO features similar to C++ except for its multiple inheritance model and its continuance of the strict strong typing of Ada83. It includes support for safety, security, real-time, systems, and other specialty areas.

As with Ada83, Ada94 has received a great deal of design critique and analysis, and is now the first standard OO language. As with Ada83, this fact of national and international standardization with supporting test suites is a great strength for the language and its users.

The choice of programming language is only one of many factors in project successes and failures. In our opinion, it is NOT the most important.

Among factors with greater influence on project outcomes, we would suggest looking at:

The language choice can do little to make up for weakness in these key areas. A mistake in any of these can kill a project. The language choice may only change expenses by a factor of two. Of course, a million dollars here and a million dollars there and pretty soon you're talking about real money.

Our data indicates that Ada has saved us millions of development dollars. For every development dollar, we could make a case for another three dollars for customer support, sales, marketing and administration costs, spent not only for the extra development but also to calm customers who are affected by our product slips or malfunctions.

Since many programming teams will not be able to switch to Ada or Ada94, Rational Software Corporation is using this data to better our own emerging C and C++ product lines. Our C code is being converted to ANSI C, which can provide some of Ada's benefits. We are trying to carry as much of Ada's value into ANSI C and C++ as we can. We are now using our own tools almost exclusively, whether for C, C++, Ada or Ada's new version called Ada94. We look forward to updating our language comparisons with C++ and Ada94 results.