An Overview of Testing

John D. McGregor

This is the debut of a column on producing quality software systems. To paraphrase Feigenbaum [Feigenbaum, 1991], Quality is the ability of a system to meet user expectations. I like this definition because it requires going beyond merely meeting requirements to meeting the needs of the user.

By way of introduction, I am an associate professor of computer science at Clemson University and a senior partner in Software Architects, a consulting firm that offers mentoring, consulting and training in object-oriented software development techniques. I divide my time among the activities of defining processes, particularly quality and testing processes, for Software Architects clients; researching techniques for testing object-oriented systems and teaching graduate and undergraduate courses in software engineering. A book I wrote with David A. Sykes, Object-oriented Software Development: Engineering Software for Reuse, was the earliest book to address testing of object-oriented software[McGregor and Sykes, 1991].

In the first few columns I will focus on the testing process as a component of the quality process. This emphasis doesnt mean that I feel you can or should try to test your way to quality. It does reflect the large number of questions concerning testing that I receive from clients, students and participants in tutorials. In later columns I will address a broader range of issues related to quality including the Personal Software Process and metrics for quantifying quality. All of these topics will be discussed in the context of object-oriented software and iterative, incremental development techniques. In this issue I want to present an overview of basic testing concerns and list issues that will be covered in the coming months.

Testing is not a technique for building quality systems; rather, it is a technique that is used because we recognize that we have failed to build a fault free system. Testing assists in its repair by identifying where the program fails to meet its specification. I will broaden the usual definition of testing that typically refers only to the testing of code. The expanded definition includes the testing of the many types of models, requirements, analysis, architectural and detailed design, that are constructed in the early development phases. These models may be represented in an executable format in a CASE tool such as BetterStateÅ or they may require a form of symbolic execution or inspection. My justification for classifying these activities as testing is that the inputs to one of these executions will be much more specific than the inputs to a typical review process (more about this below).

Using this expanded definition, there are a number of products of the software development process that should be tested including:

requirements models

analysis and design models

architectural models

individual components

integrated system code.

In fact, there should be a testing activity associated with each step in the development process. Adding the testing of models will find some faults earlier in the development process resulting in cheaper repair costs.

Although these products most often will be tested to determine their correctness, testing may be used to investigate a variety of attributes including:

performance

usability

robustness

reusability

extensibility

flexibility.

I want to divide this introductory discussion about testing into two threads. The first thread relates to what I will refer to as the testing perspective. This is a perspective from which all is questioned until proven. The second thread relates to the tester, a person whose primary role in a project is to exhibit this testing perspective. My reason for separating these two is that every developer should adopt the testing perspective sometime in their construction of a piece of software, but only select individuals will perform that role full time.

The testing perspective is an attitude that questions the validity of every product and utilizes a thorough search of the product to identify faults. This search is guided by systematic algorithms and intuitive insights. It is this systematic search that makes testing a more powerful tool than reviews and inspections. A review will almost never find something that isnt there. That is, a review typically only seeks to validate what is in the model and does not systematically search to determine if all the entities and relationships that should be there are. The testing perspective requires that a piece of software demonstrate that it is performing as its complete specification indicates that it should (and that there is no extra behavior). A product is tested to determine that it will do what it is supposed to do ( a positive test). It should also be tested to ensure that it does not do what it is not supposed to do (a negative test). This leads to the need to define the parameters of the search.

Test cases are created as part of the testing process to direct the search. A test case presents a context in which the software is to operate and a description of the behavior expected from the software within that context. In most traditional testing situations the context is an operational one in which the software is executed to determine its actual behavior. If the product under test is a model rather than actual code, the test case may contain a textual description of the context as well as some specific input values.

The testing perspective may be adopted by the same person who developed the product under test or by another person who brings an independent view of the specification and the product. Developer-led testing is efficient since there is no learning curve on the product under test, but it can be less effective because the developer may not search as exhaustively as the independent tester. The completeness and clarity of the specification becomes very important when a second person becomes involved in the testing. If a tester is not given a specification, then any test result is correct!

The specification for an object includes pre-conditions and post-conditions for each method as well as a class invariant that all facets of the object work to preserve. The pre-condition for a method documents the context that must be established before the method should be executed. The post-conditions for that method are the results guaranteed from the method if the pre-condition was true. For example, the age method for a person object might have a pre-condition that the variable birthdate <> nil. The class invariant specifies a set of conditions about the objects state that must be true at the end of every state change. An invariant for birthdate might require that its value always be between January 1, 1850 and todays date (Surely this would cover any living person!).

There are relationships among the specifications of related classes that should be reflected in the relationships among the test cases. When one class inherits from another it inherits its specification as well as the implementation of that specification. It is advantageous for the test software to also have an inheritance relationship between the test cases of the inherited class and those of the inheriting class. The advantages include improved reuse of test software and improved traceability between the production software and test software. I will describe a specific software architecture for this approach in a future column.

Testers are like good detectives. They use whatever clues are available to assist in determining where faults are most likely to occur. And then they look more intently in those places than in others. A later column will consider the role of risk analysis in setting the level of scrutiny. They conduct experiments that are intended to determine whether a system performs correctly in the specific situations described in the test cases. In a future column I will consider procedures and techniques that provide sufficiently detailed guidance to developers so that their search can be more effective.

Coverage is one metric for measuring how thoroughly the products have been tested. I will use this metric in the comparison of various testing approaches. Coverage is typically measured as a percentage of the product being tested, such as 75% of the code was executed by this test suite. Some expressions of coverage also describe the distribution of the tests over the complete product, such as one test case was constructed from each use case.

What constitutes adequate coverage is directly related to the culture of the company and industry for which the software is written. Writing software that must be scrutinized by the Food and Drug Administration or the Federal Aviation Administration requires higher quality than average and this usually translates into more comprehensive coverage of the product under test. Even when regulatory agencies are not involved, many companies have a history of doing extensive testing because reliability is a key success factor in their market. For example, telephone switching software requires high reliability and receives extensive testing.

The term tester often is used to refer to a person responsible for the system test phase in the software development process. In a later column I will discuss a number of roles that the system tester can play in the development process. I will also discuss the approaches used by system testers. Some know and are interested in what is inside the product. This inside knowledge is used to help direct the search process. Others treat the software as a black box and are directed exclusively by the specification of the product.

There are a number of issues surrounding the testing of object-oriented software systems. (Many of these issues apply equally well to procedural systems.) I will enumerate a few of them here and some of them will serve as the basis for future columns.

  1. Do we test objects or classes or use cases?
  2. What problems does information hiding present to testing object-oriented software?
  3. What techniques exist for focusing the search process on those areas that are most likely to contain faults or those areas in which the presence of faults will be most costly?
  4. What types of information are most useful in guiding the test search?
  5. What additional types of faults become possible when the software is divided into concurrent threads or distributed processes?
  6. Which metrics measure the effectiveness of the testing activity and provide insight into how to improve the testing and the development process?

Object-oriented analysis and design techniques provide for improved quality by using multiple representations that provide the ability to cross-validate among the models, such as the object, dynamic and functional models used in the Object Modeling Technique [Rumbaugh et al, 1991]. Object-oriented programming languages support information hiding that eliminate categories of the most common programming errors. With these and other quality-oriented aspects there is the possibility that object-oriented systems might require less testing than traditional systems. My experience has been that there are still plenty of faults to find and some interesting places for them to hide. I look forward to seeking them out with you.

I welcome your comments, criticisms, questions and suggestions for topics to be discussed in this space. I can be contacted at johnmc@cs.clemson.edu.

ÅBetterState is a trademark of R-Active Concepts.

Armand V. Feigenbaum. Total Quality Control, Third Edition, McGraw-Hill, New York, 1991.

John D. McGregor and David A. Sykes. Object-Oriented Software Development: Engineering Software for Reuse, International Thompson Publishing, 1991.

James Rumbaugh, Michael Blaha, William Premerlani, Frederick Eddy, and William Lorensen. Object-Oriented Modeling and Design, Prentice-Hall, 1991.

John D. McGregor is a senior partner in Software Architects and an associate professor of computer science at Clemson University. He can be contacted at johnmc@cs.clemson.edu.