Quality, Thy Name is Not Testing


John D. McGregor


Quality Assurance is a name commonly given to departments that attempt to judge the quality of a manufactured product. In typical industrial settings these departments search for defective items that arrive at the end of the production line and they watch for trends that indicate major flaws in the manufacturing process. The goal is to assure the company that as few as possible of the defective products, produced by the manufacturing process, reach the consumer. Usually their approach is to test the product and return it for repair of any defects found by their tests. For some types of products and certain types of defects, the faults can not be observed without destroying individual units. In these cases, a statistical sampling process is used to estimate the percentage of products that are defective. Determining which individual units are actually defective still requires destruction of the products in the sample.


What I would really like to see is a department of Quality Ensurance! This department would take steps to be more proactive than many "assurance" departments and to initiate activities much earlier in the manufacturing process. By the time the Quality Assurance department is in the picture, any real hope of achieving their goal of eliminating defects is gone. The need for this early intervention has been recognized by many organizations that seek to achieve ISO-9000 certification or advancement in their CMM level.


Testing is by definition a reactive activity. I canít test until the construction process provides something to examine. By then any hope of real quality is gone. I always ask students whether they feel better if testing finds lots of errors. Maybe it means that most of the embedded faults have been found. Perhaps it is just illustrating how fault ridden it really is! The fact is the defects should not be injected into the product in the first place.


In this column and the next one, I would like to discuss a series of techniques that can build in quality rather than attempt to recapture it because the techniques affect the software manufacturing process rather than the end product. These techniques are effective at least partially because they work with the standard outputs of our object-oriented development processes and thus require no new process steps.


Most object-oriented development methods make extensive use of models. The models and the process of constructing them support quality in numerous ways.

A rational context for the requirements

The use case model provides a more rational context for capturing and organizing requirements than traditional representations. Most systems have a very small set of clearly identified types of users. It is much easier to know that the complete set of users has been identified than it is to determine whether a complete set of requirements has been identified. Once an actor has been identified, it is also much easier to be certain that all of that userís uses of the system are identified.


There are several bases for capturing the requirements model and for managing the system development process. For example, I could use the commands available on the system interface as the total functionality that must be delivered. Sets of commands can then be assigned to the various delivery increments. There are two problems with this. First, defining the "system" interface may be confusing because of multiple interfaces. Second, the functionality of the system is often developed so that not every "flavor" of a particular command is delivered at the same time. In particular, exceptional cases may be delivered in the later increments.


Use cases are usually developed to a finer granularity than an interface command, at least in the scenario section, making them a more accurate unit for scheduling deliveries. The use case model also often contains what I call sub-use cases. That is I identify commonality and capture it in the use case format, even though it does not represent a true use case from an actor to an actor. This maximizes the effectiveness of the uses relation.


The actor-to-actor use cases are constructed using the uses and continues relations to connect the pieces together. When a use case is assigned to a release, the date for that release is added to every sub-use case in the actor-to-actor case. A development schedule can then be built by identifying the earliest date for which a sub-use case is required.


So why talk about this in a column on quality and testing? A failure to completely identify the use cases and to fully structure the use case model will lead to errors in scheduling development of specific capabilities. When such an error is discovered, late in development or early in integration testing, the crash effort to produce the missing functionality will lead to poor quality.

Objective basis for design decisions

Domain information can be used as the central, motivating focus of an application design. Long term company employees, including many veteran developers, will be able to judge the correctness of domain-based decisions. The rules and constraints of the domain provide an objective basis for making design decisions.


Whether a project conducts a formal domain analysis or not, object-oriented software development just naturally takes the developer into the domain. Our objects represent the stuff that domain people discuss. A recent project canceled the domain effort to save time. The application analysis process proceeded to churn and take much longer than anticipated because the development community had to form their own mental model of the domain to replace the missing domain analysis model, so much for saving time!


One of the problems with software is that it can be shaped in so many ways. We certainly have type checking as our equivalent to checking for square pegs in round holes, but larger design issues can often be handled in many ways. Domain-based models can be examined for conformance to accepted practice in the domain more rigorously than models that are based on analogies or models that are created purely for the application. Domain-based models can also be more quickly and more completely examined than models that are based on artificial arrangements of arbitrary units.


Of course this raises the issue of whether software that accurately models the domain is actually the best design. I donít propose to answer that here, partially because it really depends on the goals of the project. But my experience has shown time and again that carefully constructed domain models contribute to the quality of the software system.

Cross-validation of models

The three diagrams, object, functional and dynamic, specified by the Object Modeling Technique[4], are inter-related, but present different levels of information. UML[5] provides additional diagrams that expand the number of views available. Any one of the diagrams by itself can easily be constructed incorrectly. The process of constructing additional diagrams instantiates the inter-relationships among the diagrams making it possible to identify mistakes in the original diagrams.


A simple example of this is the consistency check between the class diagram and a message sequence chart shown in Figures 1 and 2. In the message sequence chart, each arrow represents a message, but it also represents an association between the class of the sending object and the class of the receiving object. A check should be made that there is an association in the class diagram between any two classes whose objects exchange messages. Tools such as Rational Rose do this automatically, allowing all messages to be checked.


An important aspect of the interactions among diagrams is the interaction between the class and object levels. Diagrams such as message sequence charts represent objects as do deployment diagrams. The class and collaboration diagrams represent classes. A check of the cardinality of an association can be made by a comparison of the various object-level message sequence charts and the class diagram.



Figure 1: Class diagram




Figure 2: Message Sequence Chart



Standards & Conventions

Engineers are successful, at least partially, because they systematize as much of their work as possible through the use of standards and conventions. This makes it possible to focus more effort on those tasks that still require subjective judgment. Engineers are also successful at organization. I often work with projects that have more than a hundred developers cooperating to produce a product. The development community must work as a team to produce modules that successfully integrate and compute the expected result. This integration is achieved in part because of the many standards that are established and followed.


Software engineers build products that have high complexity, like many construction projects, but are difficult to verify because they do not obey exacting quantitative laws, unlike most physical structures. A number of techniques. We rely instead on repeating successful practices. The identification, documentation and application of conventional approaches is a significant quality process.


Terminology & Naming conventions

Large projects build their own vocabulary full of acronyms and domain expressions. On one project recently two development organizations were merged to provide enough personnel to staff the project. Even though the two groups had built similar products, their terminology was very different. The analysis phase of the project took longer than anticipated because the teams had difficulty talking with each other.


The project was being built from a large framework that was being built by yet another development team. The framework teamís terminology was adopted for the basic concepts that they defined, but there were numerous naming conflicts initially since the application development team often defined a concept that was closely related to an entity in the framework. The teams needed to negotiate a set of naming conventions that identified the framework classes separately from the application classes. The size of the framework made it impractical to check each application class against a list of all classes going into a build. The addition of the namespace construct to C++ will eventually provide a partial solution to this problem as the package does in Java.


Checklists contribute to the quality of a software system by (1) bringing the experience of many people to bear on each situation and (2) standardizing the review process so that the results are independent of the participants. These lists include those items that we want to be certain are included in the system as well as those items that are most likely to contain defects.

Implementation standards/patterns

Coding standards improve system quality by guiding the original creation of designs and code. These standards represent decisions about the structure and format of the code. By following established patterns with which (s)he becomes familiar, the developer is less likely to make mistakes.


These standards encompass more than just the structure of the code. For example, issues such as the projectís use, or non-use, of multiple inheritance are often addressed by these standards. A related issue is the use, in C++, of an abstract class that allocates no memory to play the role of a Java interface. The first of these examples would be expressed as a design guideline while the second is referred to as an idiom by Jim Coplien [1]. Design patterns are a more abstract, more broadly based representation of best practice in the design of systems of certain types. Domain-specific design patterns provide a powerful means of capturing the knowledge of experts so that even novice designers can produce quality results.

Template Documents

Template documents support quality because the fixed portion of the template is carefully examined to eliminate defects. Template documents make the documentation process easier therefore it is more likely that developers will actually complete the documentation. Templates also reduce the amount of text that the developer must write. This is particularly useful in an iterative process. The danger that I have seen when templates are used is that the focus for the person using the template becomes simply completing the form as opposed to focusing on the real role of the document in the development process.

Iterative, Incremental Everything

Many words have been written about the value of an iterative, incremental process for software development so I wonít waste your time. Rather, I want to talk about applying the iterative, incremental philosophy throughout the project. For example, an iterative metrics collection and analysis process can be used to identify trends during the project and to make corrections rather than just capturing a project history[2]. A simple metric such as number of new use cases created can be used


Todayís emphasis on shortening the time to market has led to concurrent engineering efforts in requirements elaboration, analysis and development. You can either use this as an excuse for poor quality or see it as an opportunity to apply lessons learned and improve the product. For example, tracking the associations between user commands and use cases can be frustrating, even though it is useful, if the process is not automated in some way. Simple tools, such as PC-based databases or spreadsheets, are sufficiently robust to support rather large projects at an administrative level.


Figures 3, 4 and 5 illustrate two of the tables in a simple database design for tracking changes to the individual sub-use cases that together constitute the end-to-end use cases. Using the use case format that I discussed in an earlier column[3], this database supports periodic reports that produce a list of use cases, with their owners, with dependencies on the use cases that have been changed since the last report, as shown in Figure 5. We use this approach to notify use case owners that they should determine whether the changes to these use cases have resulted in a need to change the use cases that they own. In figure 4, the table defines an association between a use case and the use cases with which the first use case has a uses relation. The database also supports queries for highly critical functionality and risky functionality for use in scheduling activities in sub-releases.


This approach supports the iterative modification of the end-to-end use cases as additional system requirements are captures or as cross-validation with other models identifies conflicting assumptions.


Figure 3: Use Case Database




Assigned Release









This case considers what happens when an error occurs that requires the fault tolerant feature to be invoked.



On each boot





This case is the normal operation scenario in which there are no erros.








This case describes the invocation of the fault tolerant feature for the main processor.








This case describes the invocation of the fault-tolerant feature for a peripheral processor.




Figure 4: List of Use Cases used by other use cases









Figure 5: Changed Use Case Report

Changed Use Case

Dependent Use Case












The other techniques mentioned in this article also lend themselves to iteration as well. Template documents are intended to require a minimum of modification from one iteration to the next. The review process described below also accommodates multiple iterations.

Reviews, Inspections and Assembly Lines

The basic idea of a review or inspection of a design or document is to identify defects in the product. All too often they are seen as hurdles to be cleared rather than opportunities for careful examination. Particularly when a large number of products must be examined, a detailed inspection just doesnít happen. In fact the review process can turn into an assembly line with the products marching past inspectors who are too numb to catch most defects. Additionally, reviews seldom find anything that is missing, rather they find fault with what is there. In a later column I will discuss in more detail some techniques for improving the quality of inspections by adding a testing aspect to the process.


In a highly iterative environment, the formal process with required signatures signifying a completed document that can only be changed through a modification request (MR) either is widely violated or it becomes an overpowering burden that makes another iteration more trouble than it is worth. Alternatives include using web-based tools to allow for asynchronous reviews. The results of a study of web-based reviews found that asynchronous reviews were just as effective as face to face reviews. A second alternative is a change review in which the review is limited to those sections of the material that contain changes.


The metrics used to evaluate reviews help shape the priorities of the review process. I once heard a manager bragging about the large number of lines of code that were inspected per hour in his reviews. I have often wondered since if that manager ever decided to skip reading the code altogether since that must have been the only thing slowing him down! Ultimately the effectiveness of the review is measured by:


the number of defects found by reviews

number of defects found in the product over its lifetime.


Although this value can not be known immediately, a metric tracking program can provide approximations based on actual data collected from previous projects. Also the yield of a review can provide more immediate results:


number of defects found

number of items reviewed


where the items reviewed may be lines of code or lines of documentation or some other measure. This metric produces values such as defects per thousand lines of code or corrections per hundred pages of documentation.



A number of processes and techniques are used to create products that are of high quality even before testing attempts to locate defects. It is both more efficient and more effective to create products with quality than to attempt to add it in later. The development techniques I have discussed can and should be specialized to the exact iterative, incremental and integrative development environment found in projects using technologies such as object-oriented design and implementation and application frameworks.


Next month I intend to elaborate on one item discussed in this article, the validation of the analysis and design models for a system.


1. James Coplien. Advanced C++ Programming Styles and Idioms, Addison-Wesley, 1992.

2. John D. McGregor. Managing Metrics in an Iterative, Incremental Development Environment , Object Magazine, October 1995.

3. John D. McGregor. Planning for Testing, JOOP, February 1997.

4. James Rumbaugh, Michael Blaha, William Premerlani, Frederick Eddy, and William Lorensen. Object-Oriented Modeling and Design, Prentice-Hall, 1991.

5. Grady Booch, Ivar Jacobson and James Rumbaugh. Unified Modeling Language Userís Guide, Addison-Wesley Publishing Co., 1997.