John D. McGregor
Last month I discussed techniques for selecting system test cases. This column will continue last months theme of prioritizing elements that are to be tested, but I will shift focus to consider testing individual components. Additionally I will talk about organizing test cases. Even small systems can require a large number of test cases. Prioritization orders the test cases based on the amount of resources that will be allocated but it does not organize the cases into a logical model. If an iterative development process is being used, the problem of a large number of test cases is compounded by the continuous change of the development products. The test cases must be maintained in synch with the current state of the system; however, changes to the production system must first be tracked to the appropriate test cases.
My answer to these problems is simple: use object-oriented design and implementation techniques to manage the changes. I will show a design for organizing the test software that simplifies the task of modifying the test cases to reflect changes in the production software.
What do I mean by component when I say component testing? The short answer is, it is anything you want it to be! The techniques that I will illustrate can be used for pieces of any size or complexity. It may be a class or a cluster of classes that are tightly coupled, but it is always a conceptually atomic unit. For example, it might be the set of classes that provide the functionality of binary tree. This would include an iterator class and a link class in addition to the tree class. The intent is that the coupling among internals of this component is more intense than the coupling of the component with any external objects.
It is the component testing perspective that is important and not the size of the pieces being tested. That perspective views the software being tested as intended for integration with other pieces rather than as a complete system in itself. This both helps to determine what features of the software are tested and how they are tested.
One of the most intense arguments in testing object-oriented systems is whether detailed component testing is worth the effort. That leads me to state an obvious (at least in my mind) axiom: Select a component for testing when the penalty for the component not working is greater than the effort required to test it. Not every class will be sufficiently large, important or complex to meet this test so not every class will be tested independently.
There are several situations in which the individual classes should be tested regardless of their size or complexity:
Reusable components - Components intended for reuse should be tested over a wider range of values than a component intended for a single focused use.
Domain components - Components that represent significant domain concepts should be tested both for correctness and for the faithfulness of the representation.
Commercial components - Components that will be sold as individual products should be tested not only as reusable components but also as potential sources of liability.
Before I answer the question of how thoroughly to test, I want to summarize some of what was presented in last months column[McGregor, 1997b]. Risk analysis was applied to the task of identifying which parts of the system to test more intensely than the rest. An analysis was conducted on the requirements to determine potential business and technical risks for the development process. This analysis then mapped the risks identified at the requirements level onto individual use cases. Each use case was assigned a risk classification. All of the use cases within a category were tested to the same level of coverage. This same technique can be applied to the component level. That is, the risk classification of the use cases can be mapped onto the components. Thus not all components will be tested to the same coverage level just as not all use cases were tested to the same level.
I have already mentioned one criterion that could be used in a component-level risk analysis: whether the component is intended for reuse. The increased risk comes from the expectation that the component must respond correctly to a much wider ranger of inputs. Other criteria include the language features required for implementation of the component (using a relatively new feature such as exceptions in C++ is a higher risk), the complexity of the specification and the maturity of the development environment including the tools and the personnel.
The technique for identifying use cases that should be tested more thoroughly can be applied to the components that represent the concepts from the domain being manipulated in the use case. Once the use cases have been classified according to risk, the domain components referenced in each use case can be assigned the same risk classification as that of the use case. Of course it is seldom that simple. Since a domain object may participate in more than one use case, the risk categories for all of the use cases that reference that component must be combined to compute the risk value for the component.
Consider the example continued from last month and summarized in Table 1. Three use cases are described and assigned risk, frequency and criticality values. The table also includes a column for the test rating for the use case that was computed using the technique in last months column. The task now is to compute the test rating for individual objects given the test ratings for the use cases.
|The user modifies the name field of an existing personnel record to which they have the appropriate security authorization.
|The user saves a record that has been newly created or modified.
|The user deletes an existing record for which they have the appropriate authorization.
A component will often participate in multiple use cases that have widely differing risk values. In this example, our data management system might have a security authorization component . The security authorization component participates in the Delete Record use case that is a high risk and the Edit Name Field use case that is a low risk. Assigning a risk value to the component requires a mapping strategy similar to the one for use cases. One common conservative strategy is the maximum value strategy. In the case of security the maximum risk associated with any of its use cases is a value of high so the component is given a value of high. Table 2 illustrates this further. There is a column in the table for each use case and a row for each class. The test rating for the use case is entered in those rows that correspond to components that participate in satisfying the use case. The Name string and Security authorization components participate in the Edit Name Field use case. The mapping strategy is used to produce the entry in the final column for each component. An alternative strategy is to average the risk values.
|EDIT NAME FIELD
Table 2 - Risk Allocation to Components
Over the last couple of years, we developed the Parallel Architecture for Component Testing (PACT) [McGregor & Kare, 1996]. It is an architecture for the software needed to support the component testing process. The goals of the architecture are to organize the test cases used to test a set of classes, facilitate the reuse of those test cases, and improve traceability between the production software and the test cases.
The architecture begins with a set of abstract classes that encapsulate the basic functionality needed by all the test cases. For example, the result of a test case execution is usually logged to a file or database. A basic connection to this repository can be created in one of the generic classes and then inherited by classes that refine the functionality for specific types of test cases.
//General constructor and destructor
//These are the three main test suites. They are
//listed in the order of priority. They must be
//implemented for each test class.
virtual void functionalSuite()=0;
virtual void structuralSuite()=0;
virtual void interactionSuite()=0;
//These are utility methods that are public because
//they allow the user of the test class to select
void wait(long); //Used to pause output that is directed to the screen
void showheap(); //Prints the size of the heap; used to check for memory leaks
//Prints the number of test cases that failed
//This method prints a test report header on the stream
//This method reports when the class invariant method has failed.
//This method is defined for every class. It may
//be constructed from a series of smaller methods so
//that the pieces may be used down an inheritance hierarchy.
virtual boolean classInvariant()=0;
//These are utility methods that provide a uniform
//method for reporting success and failure.
virtual void reportSuccess(int);
virtual void reportFailure(int);
Figure 1: PACT Abstract Class Specification
The abstract class in Figure 1 is a root class for a hierarchy of test classes. For each production class we have decided to test as an independent component, a test class is created. The test class is created by inheriting from this abstract class, either directly or indirectly. The new test class inherits directly from the abstract test class if the production class it is intended to test does not inherit from any tested class<1>. The test class inherits indirectly by inheriting from an existing test class. This occurs when the production class about to be tested inherits from an existing production class for which a test class is already defined. This is where the parallel nature of the architecture is seen, as illustrated in Figure 2. I am showing the simplest example where each component is a single class.
Figure 2: The Relationship between the Production and Testing Architectures
Other abstract classes provide access to required services such as hardware with which the program under test must interface and support software such as testing tools. Complex or embedded systems often need to establish a specific state in the encompassing hardware prior to running certain tests. This is straight forward to accomplish with PACT. I have used PACT classes to encapsulate commercial tools that compute the complexity of a piece of code and to standardize the handling of exceptions.
There are two aspects of this approach that need further explanation. First, we have found that representing test cases as individual methods that belong to a test class is the correct level of granularity. As an alternative, it is possible to represent each test case as a separate object. This introduces the possibility of maintaining state about each test; however, the results of a test are often simply reported and not held for further analysis. The use of separate objects would support this delayed analysis but at the cost of additional management overhead.
Second, having the inheritance structure of the test classes parallel the inheritance structure of the production classes improves maintenance. One of the ways to understand the architecture of an object-oriented program is to understand the inheritance structure. If the inheritance structure is the same in the test software as it is in the production software, a developer that understands the production software will more quickly understand, and be able to modify, the test software. When a change is made to the production software, locating the place in the test software to make the corresponding change is made much easier.
The advantages of PACT reach beyond the ease of management and maintenance. If the system under test is a framework intended to be provided to numerous development groups, the PACT software can be handed to the application developers along with the production software. As the framework is evolved into a specific product by modifying framework classes or by adding additional classes, the test classes can easily be modified or created.
PACT has been implemented in several languages and in a variety of settings. It has been implemented using C++, Smalltalk, Java and SOM. The projects using PACT have had from 4 to 1500 developers and have ranged from business to highly technical domains. Anecdotal reports and personal experience have verified that PACT saves time and improves the quality of the test code.
Component testing is an important aspect of the test plan for a project. A very large percentage of errors can be found at this level, but not without a cost. One technique for managing this cost is to test different classes to different levels of coverage. A second technique for managing the cost of component testing is the use of a specialized software architecture that optimizes the reuse of the test cases.
The PACT provides the basis for an extensible testing framework. As such it is a natural addition to a complete framework product. Just as an application framework is intended to speed the development of an application within a specific domain, the testing framework facilitates developing the software required to test the components of the application framework. Next month I will provide more details about PACT and its role in component testing.
<1> This means that for languages such as Smalltalk, we are not going to construct test class all the way up to Object.