In the last issue I introduced the Parallel Architecture for Component Testing (PACT) as a way of supporting testing components and particularly testing them to different levels of coverage. This month I will provide additional details about PACT and in the process will explore additional issues related to component-level testing.
PACT is not a testing technique, rather it is an approach for organizing the software that implements test cases. PACT does not prescribe specific levels of test coverage nor does it provide guidance in what tests to construct. PACT is a software architecture that defines the structure of the test software. The objectives of this architecture are to minimize the effort required to build and to maintain the individual test cases. The architecture is independent of testing tools that are used and is applicable to a wide range of implementation languages.
One of the issues that will be discussed below is separating the test code from the production software. When I refer to test code, I am talking about the software needed to test the application under development. I am not referring to code which might be considered self-test code that is part of the expected functionality of the delivered application. The self-test code will be tested the same as the rest of the application.
The PACT specifies two basic patterns for the testing software:
(1) There is a class in the test software for every component in the production software judged to be sufficiently significant to be tested independently. What is judged significant will vary from one project to another based on how critical the component is to the success of the system. The techniques for risk analysis discussed in the previous columns can be applied here to determine which components to test.
(2) For a new production class, its test class inherits from the test class of the production classs superclass. This is the parallel nature of the architecture.
The rationale for the architecture addresses several issues:
(1) Reuse of test code - The inheritance relationship between test classes for production classes that are also related via inheritance facilitates the reuse of individual test cases.
(2) Separation of production and test code - There are several reasons for keeping the two types of code separate. First, the production code remains smaller and thus less complex. Second the executible also remains smaller requiring fewer resources. Finally, these two pieces are sometimes written by different groups and the physical separation becomes necessary.
(3) Maintenance of test cases - In an iterative development process, the tests should be easy to apply repeatedly across the development iterations. They must also be easy to maintain as the production code changes. The inheritance relationship in an object-oriented language supports the development of code with these characteristics.
Lets briefly consider two design alternatives and compare them to the PACT.
(1) The test code is included within the production code. The most significant problem with this approach is that the executible becomes larger than necessary due to the test code. The obvious solution of removing the test code after testing, either by commenting it out or setting a pre-processor variable, leads to the equally obvious problem of invalidating all the tests that have been conducted!
(2) The test code is included in a class that is a subclass of the production class being tested. This approach overcomes one of the difficulties with the first alternative: the test code is now separate from the production code. The problem is that unless the implementation language supports multiple inheritance there is no easy way to reuse test cases.
The precise structure of the PACT structure is determined by the language being used to implement the product. The test class needs access to the internals of the production class. In C++, we have declared the test class to be a friend class to the production class. This provides the test class with complete access and does not affect the quality of the production code. In Smalltalk, the instVarAtName method in the Object class allows access to the dictionary that contains all of the member attributes of an object. In most object-oriented languages there is some construct that allows special access.
The typing system can also affect the implementation of PACT. In most applications the typing system of C++ allows the substitution of objects from later in the production and test hierarchies for ones earlier in the hierarchies. However, there can be situations when the overriding of a test case requires casting of a test result.
In this section I am going to present a very small example. I take a risk in doing this because the benefits of PACT are realized when a number of related test classes must be created and when tests must be maintained and applied repeatedly over development iterations. Use the example only to understand the structure and basic approach.
Listing 1 presents the interfaces for a two class inheritance hierarchy that implements two simple timer classes. These are the CUTs (Classes Under Test). In the subclass, the start and read_elapsed methods are overridden, but the other methods are inherited as originally defined. The test cases for all of these methods are inherited unmodified as well.
//allows the test driver access to the internal value
//class invariant: stop_time - start_time >= 0;
//post-condition: An instance has been created and is
//in the idle state.
//pre-condition: An instance of SimpleTimer exists.
//post-condition: An instance has been deleted from memory.
//pre-condition:The instance is in the idle state.
//post-condition:The instance is in the running state.
//pre-condition: The instance is in the running state.
//post-condition: The instance is in the stopped state.
//post-condition: Value of the timer has been returned.
//pre-condition: The instance is in the stopped state.
//post-condition: The instnace is in the idle state.
//start_time is set when the timer is started
//stop_time is set when the timer is stopped
//running is true if the timer is accumulating time
//running is false if the timer is either stopped or idle
class RestartableTimer : public SimpleTimer
//Post: An instance of timer exists and is in the idle state
//Pre: Instance of RestartableTimer exists
//Post: The instance no longer exists
//Pre: Instance of RestartableTimer exists
//Post: The timer is in the running state
//Pre: Instance of RestartableTimer exists
//Post: Returns 0 if the timer has never been started or
//Returns the time that has accumulated for all the time
//the timer has been running
//Stores the time that has elapsed in previous times
//when the timer was in the running state
The foundation of PACT is a set of abstract classes that provide basic functionality that can be inherited into each concrete test class. Listing 2 shows the interface of one such class. The services provided by the abstract classes include common exception handlers and common input and output facilities.
//General constructor and destructor
//These are the three main test suites. They are
//listed in the order of priority. They must be
//implemented for each test class.
virtual void functionalSuite()=0;
virtual void structuralSuite()=0;
virtual void interactionSuite()=0;
//These are utility methods that are public because
//they allow the user of the test class to select
//This method prints a test report header on the stream
//This method reports when the class invariant method has failed.
//These utility methods link the PACT classes with a
//standard testing tool, in this case the LDRA testbed.
//These two methods invke the static and dynamic
//analysis facilities in LDRA and apply them to the
//This method is defined for every class. It may
//be constructed from a series of smaller methods so
//that the pieces may be used down an inheritance hierarchy.
virtual boolean classInvariant()=0;
//These are utility methods that provide a uniform
//method for reporting success and failure.
virtual ostream* reportSuccess(int);
virtual ostream* reportFailure(int);
//These methods are handlers for the standard system
//exceptions. These can be inherited and reused but
//the actual catch clauses cannot be inherited and
//must be constructed for each test class.
void xmsgException(const xmsg&);
void xallocException(const xalloc&);
The concrete PACT classes provide the individual test cases and the interface is not very informative. Therefore Listing 3 provides the implementation of a test case and the encompassing test suite. Each test case is divided into three parts. The first segment ensures that the OUT is in the appropriate state to begin the test. The second segment is the sequence of messages that constitute the test, which in this case is to exercise the read_elapsed method. The final segment verifies the result and/or logs the information for later examination.
The test script for this test case is very simple. It constructs the object under test (OUT), administers the test and then cleans up by deleting the OUT. A more industrial strength test script might open files, configure hardware and perform other types of environmental maintenance.
*ostr << "***TEST CASE 5" << endl;
//Put OUT into idle state
//Start timer and read elapsed time
time_val = OUT->read_elapsed();
*ostr << "+++ET is - " << time_val << " (read_elapsed)" << endl;
OUT = new CUT;
To fully describe the use of PACT I want to use a generative pattern language<1>[Coplien]. I will greatly abbreviate each pattern description but the complete language is available from http://www.cs.clemson.edu/~johnmc/PACT. An earlier version has also been published[McGregor]. I am very grateful to Anuradha Kare for her assistance in developing this language.
Each pattern will be described in three parts. The first part will present the situation that exists that must be solved and will provide a brief discussion of trade-offs. The second section concisely states what action should be taken to resolve the design problem. Finally, the third section presents the new context that results from applying the pattern. This section will also direct the reader to other patterns that can now be applied.
Context & Forces: In an environment where components are to be reused, they must be tested separate from any client application. There can be a rather large set of test cases which must be managed and, be cause we assume an iterative development environment, the test cases must be applied multiple times over the life of the development process.
Solution: Each component that is of sufficient size or value to test will have an accompanying class that encapsulates all of the software needed to test the production component.
Resulting context: It is easy to identify the test code that corresponds to a specific production component. Changes to the production code can be reflected quickly in the test software. The classes of test cases can be developed efficiently to minimize overhead. One technique for this is to Create parallel hierarchies. The test class allows the tester to Sequence test cases by using test scripts.
Context & Forces: The project is using object-oriented development techniques because the relationships among objects such as inheritance and aggregation encourage and support reuse. This results in an architecture for the application that uses abstraction. These relationships also result in more pieces that must be managed and maintained.
Solution: Organize the test classes into inheritance and aggregation hierarchies that correspond to the inheritance and aggregation hierarchies in the production code.
Resulting context: The test software is organized around the same architecture as the production code. I use the term parallel because the architecture of the test code never coincides with the production code but it always stays the same distance away. If a developer studies the production architecture, they automatically understand the architecture of the test software. This organization of the test classes makes it easier to Test a method in the context of a class and to Overcome information hiding in order to observe state.
Context & Forces: Traditional unit testing techniques focus on individual functions. This testing is often informal. Each unit test would build a scaffolding to support the testing of its function. In our case, this would duplicate much of the environment defined by the class and realized in each object.
Solution: Plan tests around classes rather than individual methods.
Resulting context: The class is the unit of test rather than an individual method or even an individual object. Fewer stubs are needed since all of the methods of the CUT are available in their entirety. A class definition may define an infinite set of objects so the class definition must be tested by sampling from the community of objects. To test an object we must Overcome information hiding in order to observe state.
Context & Forces: The test class must be able to see inside the objects from the CUT. Otherwise the test code will have to rely on the accessor methods in order to verify the results of a test. Placing the test code inside the CUT would achieve this objective but it would increase the size of the application code.
Solution: Use a language-specific mechanism that bypasses the information hiding feature of the implementation language, but that does not affect the application code. The class definitions in Listing 1 each have declarations that make the test class a friend of the production class.
Resulting context: Test code can be written that can directly access the state variables of individual objects to verify results. This direct verification can create undesirable implementation dependencies between the test code and the production code and so should be limited as much as possible. This direct access to an objects state variables leads to the need to Create a baseline test suite.
Context & Forces: When the test code has visibility inside the objects under test, direct access by the test code to the production code increases the maintenance requirements on the test software. Another alternative is for test cases to rely on untested accessor methods.
Solution: Create a baseline test suite that tests all of the accessor methods specified in the class. This suite should be executed prior to other test suites.
Resulting context: This test suite uses direct access to the variables within an object to test its accessor methods. For the classes in Listing 1, a baseline test suite would test the read_elapsed method. The changes to test cases, brought on by changes to the production code, are minimized. This suite allows other tests to use the accessor methods with confidence in the results they produce.
Context & Forces: A number of test cases that test individual methods have been identified. To test the interaction of methods we wish to execute a sequence of methods and then evaluate the results. Functional tests execute scenarios to determine whether a specific method performs to its specification. When various protocols with other classes are tested, the scenario definitions can be combined into sequences.
Solution: Use test scripts to provide an environment in which one or more test cases can be applied to the same OUT. Listing 3 shows a simple test script.
Resulting context: By separating the scenario definition (the test case) from the administrative details (the test script) of creating the object under test (OUT) , the scenario definitions can be combined in a number of ways. Individual test cases can be referenced multiple times within a single test class. These scripts provide test cases with the ability to Record message outcome for further analysis.
Context & Forces: There are several different viewpoints from which a component can be tested: specification, structural, performance, etc. Some test cases can be used to test from multiple viewpoints.
Solution: Use test suites to group test cases and scripts according to their purpose.
Resulting context: The appropriate testing for each specific purpose is defined early in the development process and then repeatedly applied whenever the need arises. For example, the regression test suite is a sampling of tests that can be used to quickly point to specific types of testing that are required after a development iteration. Listing 2 specifies test suites that application-specific test classes must define.
Context & Forces: The test code stimulates the OUT which in turn produces various outputs and state changes. These results must be examined to determine whether the test was successful.
Solution: Provide techniques such as streams to capture all of the values output by the production code under test. The usual output will be the objects serving as return values from methods including any exceptions.
Resulting context: The abstract test classes will define capabilities for logging results as simple as pass/fail or an exception object that was unexpectedly encountered. In the example code in Listing 2, a stream is used as the technique for logging information. One such action is to Handle exceptions. By capturing this information we are in a position to Verify results.
Context & Forces: The test code must provide the environment that would be provided by the production code, that is the client of the code under test. This includes catching any exceptions that are thrown, whether they should be or not.
Solution: Provide generic exception handlers as methods in the abstract classes and specific exception handlers in the application specific test classes. If the implementation language defines a set of primitive exceptions, the abstract test classes should define handlers for each.
Resulting context: The test classes now provide the environment necessary to capture exceptions produced during execution. In Listing 2 handlers for two primitive C++ exceptions are specified.
Context & Forces: Once the outcomes of the test cases have been recorded, they should be evaluated to determine whether the result is correct or not.
Solution: Write each test case to evaluate as much of the results of its execution as is possible. This may be as simple as the comparison of two integers or as complex as a bit by bit comparison of two bitmaps.
Resulting context: By automating each test at its source, the overall verification will be more accurate and it will be easier to make certain that the test code is correct.
The PACT provides a structure within which the tester can build the test software in an efficient, effective manner. Our experience in a variety of projects has validated the benefits of PACT. Over the long term, PACT provides economies in terms of the amount of code that must be written, the effort required to apply test cases and to evaluate the test results.
The software built around the PACT can become an important part of a product. I have many clients that have developed test code based on PACT to test frameworks during in-house development. I can also see the test code being shipped with frameworks to the clients who will in turn use the test code as the basis for creating their own test code for the application-specific additions to the framework.
Jim Coplien Generative Pattern Languages, C++ Report, July/August 1994.
John D. McGregor and Anu Kare. A Framework for Testing Object-Oriented Components, 13th International Conference on testing Computer Software, June 1996.
<1> A generative pattern language is a set of related design patterns in which the context resulting from applying one pattern establishes the context in which some subset of the patterns becomes eligible for application.