Our Blog Excerpts Savings Contact

logo

Dorset House Publishing
High-Quality Books on Software Engineering and Management.  Since 1984.
dorsethouse.com > features
Features       Excerpts       Interviews

 

iDH Sign-Up


Get Our e-News
Delivered by FeedBurner

A Methodology for System Testing

by Nathan Petschenik

Adapted from Chapter 12 of System Testing with an Attitude. Reprinted by permission. All rights reserved. See below for copyright notice.

This chapter introduces the system-testing strategy and methods that I have used and refined over some twenty years. The methodology is presented first in outline form, with subsequent chapters used to provide detail for its application. To facilitate cross-reference, I use a capital-letter "M" to make it clear when I am referring to the Methodology described in this book.

Presenting the Methodology will require that some terms be defined, but many of the terms won't come to life until later in the book, so please be patient. In addition to providing definitions, I need to make two initial points to frame the Methodology and its relationship to the chapters that follow:

  • Applying the Methodology requires individuals to have an in-depth understanding of the users of the system to be tested, the business environment in which the system will operate, and the risks associated with the system. This subject is the focus of Chapter 13: "Understanding the Typical User."
  • Steps in the Methodology are to be applied throughout the entire software development life cycle (not just during the system-test phase of the project). Chapter 19: "Game Plan for a System Test" discusses when to apply each step in the Methodology in the context of the SDLC.

The objective of the Methodology is to provide a framework for developing well-documented, repeatable, data-dimensional system tests that cover typical business flows. I'll start by defining what I mean by the terms "well-documented," "repeatable," and "data-dimensional" (I define what I mean by "typical business flows" later in this chapter and in Chapter 13: "Understanding the Typical User"). Once you become familiar with my definitions, you will see that these terms apply to characteristics needed for other types of testing, not just system testing.   [top]

The techniques I advocate for achieving these characteristics have broader application than merely to system testing, but aspects of the Methodology that are specifically pertinent to system testing are the architecture of system tests and the story of the test. These two aspects are detailed in Chapter 14: "Defining an Architecture of System Tests" and Chapter 16: "The Story of the Test," respectively. Let's look at the definitions.

A test is said to be well documented if and only if

  • the documentation for the test clearly identifies the objective of the test, which includes classifying the specific types of problems that the test is designed to detect.
  • the test documentation clearly identifies the expected results of the test and how to tell whether the test passed or failed by comparing expected results with actual results.
  • the documentation is sufficient to allow someone other than the person who developed the test to execute the test with exactly the same results.
  • the documentation is sufficient to allow the test to be maintained successfully by someone other than the person who developed the test.

A test is repeatable if and only if

  • the test will produce exactly the same results as long as the documented preconditions of the test are met and the system capabilities have not changed. This means that each time the test is executed correctly, the expected results and the actual results will match as long as the precise version and configuration of the system under test are exactly the same.

A test is data dimensional if and only if

  • variations of the test have been analyzed and action has been taken to expand the scope of the test to cover additional, important data situations efficiently.

Why are these characteristics so important to testing? The value of developing well-documented tests should be pretty clear from the definition. Well-documented tests are designed to become the lasting, intellectual property of the test team. Their value in protecting against a clearly defined class of problems is not tied to the specific developer of the test. The developer of the test can move on to other jobs without fear of being needed to execute or maintain the test. Even while the test developer remains with the system-test team, a well-documented test (in the way we have defined it) makes it possible for a less experienced (and less costly) test-execution specialist to execute the test after it has been fully debugged, freeing up the test developer for more productive tasks.   [top]

As for repeatability, achieving this characteristic makes a test straightforward to execute and re-execute since the test works exactly the same way each time. This increases the likelihood that the test will be executed correctly even by a relatively inexperienced test-execution specialist or, even better, by an automated test tool.[1] But whether manual or automated, repeatability helps to ensure that the test will have lasting value. Rather than being a "one-shot throw-away," a repeatable test is designed to be executed and re-executed multiple times in the testing of a particular software release and to become a regression test in the testing of future software releases. Finding problems the first time a repeatable test executes is only the beginning of its value. The same test can be used again to check out the fixes to the problems that were found. After that, the test can be used to check that further changes to the system do not inadvertently destroy old, existing capabilities in the current system under test, and in future releases of the system under test as well.

As for the third characteristic, making a test data dimensional is a more-bang-for-the-buck item. It means that, as an integral part of developing the test, possible variations are considered that could extend the range of problems that the test protects against. It is often the case that, while a test is being developed, the test logic can be extended to cover additional data variations with little cost and without complicating ongoing maintenance. Sometimes this involves making the test data-driven and sometimes it doesn't.[2] We'll discuss these characteristics (and how to achieve them) in subsequent chapters.

Following are additional terms I use in the Methodology, along with their definitions:

Basic-sanity test: A system test designed to detect cases in which the system has not been installed correctly and/or to quickly identify problems that prevent fundamental capabilities of the system from being accessed.

Administrative-functions system test: A system test designed to find problems that would prevent an administrative user from performing work activities associated with such functions as adding/changing/deleting reference data, system default settings, security, and authorization permissions.

Installation/migration system test: A system test designed to find problems that would prevent the initial installation of the system from being done correctly. Installations often include migrating data from an earlier version of the system, a system being replaced by the system under test, or from manual records.   [top]

Flow-thru system test: A system test in which system components are strung together and exercised in a complete and meaningful business context, causing interfaces to occur naturally (rather than simulating them) and ignoring any intermediate results that are not visible to users. Flow-thru tests are the result of implementing "test stories," described in Chapter 16: "The Story of the Test."

Flow-thru support-system test: Also called a business-flow or business-process support test, this is a system test whose objective is to prepare the system-test environment (to create data situations, for example) for specific flow-thru tests that expect these conditions to exist at the time that these flow-thru tests begin to execute.

Special-situations system test: A system test that is designed to supplement one or more other system tests by protecting against less typical situations than are covered by the other test(s). A high-risk special-situations system test also protects against problems not covered by other tests; however, it focuses on protecting against situations that have been identified by risk analysis as having highly disruptive (or worse) symptoms even though the conditions under which the problems might be triggered may not be particularly likely to occur.[3]

Higher-order system test:[4] A system test that is designed to protect against problems with performance and volume characteristics of the system (as opposed to functional characteristics). For example, a load test is a higher-order test designed to detect problems that show up only when the system is subjected to the level and types of traffic that the system might realistically encounter in a production operation. A volume test is designed to protect against problems that show up only when the system is processing the high amount of data that a system might typically encounter in a production environment. A stress test protects against problems that could occur when the system is pressed beyond its design limits in terms of traffic or data volume.

With these definitions in mind, we are ready for the Methodology, which is presented in the balance of this chapter and boxed in gray in excerpts as they appear in subsequent chapters. I will amplify and explain aspects of the Methodology in the chapters that follow.   [top]

Methodology for System Testing

Framework for the Methodology

Effective planning for an architecture of well-documented, repeatable, data-dimensional system tests begins with identifying a set of typical business flows that reflect the usage of the system to be tested. A typical business flow is a description of how users will accomplish work through a sequence of interactions that one or more users (or other systems, devices, and so on) will have with the system. Typical business flows (sometimes simply referred to as "business flows") should be identified and documented as part of the requirements/specification process.[5]

In addition to understanding the business flows of the user, the Methodology requires that risk analyses be conducted at strategic points in the system development life cycle. Understanding the risks associated with the functionality of the system is necessary so that appropriate priorities can be assigned for developing the system tests in the architecture. Knowing the relative importance of tests in the architecture will influence the order in which they will be developed (and may determine whether or not a system test will be developed at all should a resource shortfall occur).

The Steps in the Methodology

1. Follow Steps 1.1 to 1.5 to define an architecture of system tests for the system to be tested. Each node in the architecture must be assigned a unique identification.

1.1 Begin by allocating a node in the system-test architecture for each business flow identified and documented during the requirements/specification process. Add a node for each of the basic-sanity tests and higher-order tests. Be sure that there are nodes in the architecture that correspond to administrative functions and to installation/migration (if not, add these nodes and decompose them to correspond to the functionality for these areas specified in the requirements).

1.2 For each node corresponding to a business flow, decompose the node into three sub-nodes: flow-thru, high-risk special situations, and other special situations. (The high-risk special-situations node is for system tests that are not part of a typical business flow but are important based on risk assessment.)   [top]

1.3 For the administrative-functions and installation/migration nodes in the test architecture, decompose each node into two nodes: a flow-thru support test and a special-situations test (high-risk special situations involving administrative functions and installation/migration should be covered in the flow-thru tests associated with business flows).

1.4 Go through the requirements documents and systematically determine where in the architecture of system tests each requirement is covered. Document this coverage in a traceability matrix. Coverage should be based on the following criterion: The requirement is covered by a specific system test if the system test would be responsible for finding that the requirement is missing or erroneously implemented, assuming that the system test fulfilled its objective perfectly. If you cannot find a system test that meets the requirement, define an additional node in the architecture corresponding to the test that covers the requirement. If any node in the architecture appears too broad in scope for a single system test, split the node into multiple system tests. If you can conceive of any type of problem that could occur with the system for which there is not a test in the architecture responsible for protecting against that type of problem, add a node to the architecture representing a system test whose objective would include finding that type of problem (I call this type of placeholder system test a "worry bead"). Keep iterating until all requirements have been covered by one or more tests in the architecture of tests, until the scope of each node appears manageable, and until the architecture appears conceptually complete from a problem-coverage standpoint.

1.5 Document the test objective of each node that has not been further decomposed. The identification of requirements and hypothesized types of problems that are covered by a specific system test should be included as part of the documentation of the test objective.

2. Prioritize the tests in the test architecture and determine the sequence in which the tests are to be designed and implemented.

3. For each basic-sanity test to be developed, design and document a set of test cases (action plus expected results) that can serve as part of the entrance criteria into system testing. (See the documentation principles described in Step 4.7.) The objective of this test should be to detect situations that indicate that the installation of the system has not been done correctly and to quickly identify problems that prevent fundamental capabilities of the system from being accessed. If possible, these tests should be designed to be independent of the data environment at the start of the test and should leave the data environment unchanged at the end of the test (if a clean-up of the data environment is necessary, you can make the clean-up part of the test). This allows you to execute the tests immediately after an installation of a new iteration of the system and, if the installation appears to function properly, to begin execution of other tests immediately after the basic-sanity test has executed.   [top]

4. For each system test to be developed that corresponds to a business flow, follow Steps 4.1 through 4.7.

4.1 Construct the story of the test -- that is, a story that illustrates how users will execute a typical instance of the business flow with typical data. The story of the test has three parts:

  • Preconditions: This is the "once-upon-a-time" section of the story. It includes actions and/or assumptions regarding the functional state of the test environment at the time the test begins. This includes the results of actions that may have taken place before the system under test is installed and operating (for example, it may describe transactions that took place prior to the data migration to the new system). As described in the steps below, this part of the test may be affected by the technique chosen to solve cycle-acceleration and repeatability problems. Documentation for this section of the test (see Step 4.7) should introduce the characters in the story (and other systems that may be involved in the test) and describe the state of the system at the time the body of the test begins.
  • Body of the test: This is the "plot" section of the story. The plot begins after all the actions in the once-upon-a-time section are complete. The plot describes the interactions between the characters and the system under test to get business done through the capabilities of the system. These interactions compose the test cases through which the plot unfolds. Each test case must have a unique ID and must include expected results for actions (see Step 4.2). As test cases are added to cover specific requirements, the mapping between the requirement, the test, and the test case is recorded in the traceability matrix. A test case is said to "cover a requirement" if the test case protects against the requirement being missing or erroneously implemented. The identification of requirements and hypothesized types of problems that are covered by specific test cases in the system test should be included as part of the documentation of the test (see Step 4.7).
  • Post-conditions: This is the "happily-ever-after" section of the test. It includes actions and/or descriptions that affect how the test leaves the test environment at the test's conclusion. (As described in the steps below, this part of the test will be impacted by the technique chosen to solve repeatability problems.) Documentation for this section of the test (see Step 4.7) should describe both the state of the system at the time the test ends and whether the test is serially rerunable and/or capable of concurrent execution (see Step 4.4).

The story of the test becomes part of the system-test documentation (see Step 4.7).

4.2 Identify what specific system responses need to be observed to determine if the actions performed by the characters worked as expected (focus on the variables, not the values, at this point). As part of the plot section of the test, be sure to have the characters observe these and only these results using capabilities provided by the system wherever possible. Using as much information on the implementation of the system as available at this point in the life cycle, figure out how the expected values should be determined and record this information as part of the documentation of the system test (see Step 4.7).   [top]

4.3 Solve cycle-acceleration problems. Perhaps a test story is intended to illustrate events in a business cycle that would normally take place over an extended period of time. In order to go through a complete cycle in the test story, you may need to reach a threshold (for example, sell out a sporting event) or move through events that happen only at specific time intervals (daily, weekly, monthly, or yearly, for example) in the real business environment. These types of tests pose problems because (obviously) it's unrealistic to spend a year running a test that illustrates activity that takes place over a period of a year (or similarly, of a month, a week, or even a couple of days). In the case of needing to reach a threshold in a test story, find creative ways to perform the necessary activities in a way that avoids unnecessary tedious and redundant effort but still accomplishes the objectives of the test.[6] In the case of test stories that require cycle acceleration with respect to time, if the software you are testing has been designed to address this testing issue, it may be possible to control the passage of time parametrically or by changing data fields through capabilities available to the system-test team.[7] (Watch for this testability issue in reviewing system designs for testability.)

4.4 Solve repeatability problems (so that the system test will execute exactly the same way each time it is executed, assuming that the software configuration of the system under test remains invariant). In particular, solve the following problems:

  • The "remnants" problem: Once you've executed the system test, the data in the test environment will change; the results of rerunning the test may be different, unless you take action to clean up the "remnants" of the last execution. In some cases, it does not take much work to make the test serially rerunable; in other cases, the data must be refreshed to an initial state. In any case, find ways to address the remnants problem.
  • The "common-sandbox" problem: You will probably be sharing the test-database environment with other system testers. You need to be sure that the impact of their testing will not change the results of your testing (and vice versa). In other words, tests should be pair-wise non-interfering.
  • The "self-competition" problem: If two or more copies of the test are to be run simultaneously (for load, performance, or stress testing, for example), you need to be sure that the test will not interfere with itself. This is essentially the common-sandbox problem, except that the test is sharing the environment with itself. In some cases, it doesn't take much work to make the test capable of concurrent execution. For some system tests, it may be too costly or too difficult to solve this problem completely. If the latter is the case, the documentation should indicate the restrictions or special issues associated with concurrent execution of this test. Your method for addressing these repeatability problems will impact the once-upon-a-time section and/or the happily-ever-after section and should be documented there.

4.5 Determine if there are important situations in the life cycle of the business flow that could be in progress when the software of the system under test is actually installed and operating. If this is the case, design additional plot lines with unique test-case IDs that correspond to these transactions in progress. Reuse elements of the plot that you have already designed in Steps 4.1 through 4.4 whenever doing so is useful. Enrichments will generally need to be made to the test-data environment in order to realize these plot lines. For example, the once-upon-a-time section may require that you have data in the test environment that correspond to paper or electronic records that preceded the installation/migration of the system under test. Review cycle-acceleration issues and repeatability issues for the new plot lines and make adjustments as necessary.   [top]

4.6 Identify variations of the test story that could expand the coverage of the system test, possibly by changing the selection of data used by the test. If appropriate, incorporate these variations using the following techniques:

  • Spread data variations among multiple characters in the story.
  • Add more characters to the test story that exhibit the variations.
  • Add test cases to the special-situations test.
  • Add a new test (or tests) to the architecture of system tests that is (are) responsible for covering these variations.

In some cases, it may make sense to parameterize the test so that you can cover a variety of important data situations by assigning different values to the parameters and rerunning the test.[8] Parameterization is generally preferable to "cloning" the test logic, which complicates system-test maintenance. If you parameterize the test but you decide that an individual variation should be executed independently of the test being developed, add a separate node to the system-test architecture for that variation. If you decide that a particular variation should be part of the scope of the test but you will not develop that variation immediately, make a note of this in the documentation of the test objective.

4.7 Develop the system-test documentation for the test. Define the objective of the test by describing the types of problems this test should protect against and by identifying the specific system requirements that are covered by the test. A requirement is covered by a system test if the system test contains test cases that would be responsible for finding that the requirement is missing or erroneously implemented, assuming that the system test fulfilled its objective perfectly. Hypothesized problems are covered by a system test if the system test contains test cases that would protect against this problem reaching a user, assuming that the system test fulfilled its objective perfectly. Include any information in the test documentation that would be needed if someone other than you will maintain the test. Likewise, include any information in the test documentation that would be needed if someone other than you will execute the system test. Document the three parts of the story of the test, the technique for achieving repeatability, the solution to the cycle-acceleration problems, and parameters that would need to be changed to realize important variations identified in Step 4.6. Select the values of data for the test using typical values from the operating range. Document the script of actions and, based on the user documentation (or equivalent information), document the expected results (see Step 4.2) that implement the system test (the details of the plot). If there are variations of the data that you consider important and within the scope of this test but that are not implemented in the current version of the test, document these variations as an aspect of the test objective that is not achieved in the current version.

5. Design and document the flow-thru support tests for both the administrative-functions node and the installation/migration node in the test architecture. These are the tests that create the data environment that provides the starting point for the flow-thru tests (either through installation/migration activity or by reference-data and security-data updates). Use the once-upon-a-time sections of the flow-thru tests as the basis for determining what belongs in these flow-thru support tests.   [top]

6. For each high-risk special-situations test that is to be developed, design and document test cases whose objective is to protect against problems associated with the high-risk special situations that have not already been covered by other tests that have been developed.

7. If the architecture includes higher-order tests (load tests, for example) that need to be developed, try to realize these tests by building on other tests in the architecture that have already been fully debugged. In order to avoid the need to maintain multiple versions of the tests being reused, avoid cloning them. In other words, create load tests, stress tests, and performance tests by leveraging debugged business-flow tests as a basis of representative, typical-user activity on the system.

8. For each special-situations test that is to be developed, design and document test cases whose objective is to protect against problems associated with the special situations that have not already been covered by other tests that have been developed.

9. Once the system is available for system testing, install the system, performing any data migration called for in the installation instructions provided by the development team. Use the flow-thru support tests for both the administrative-functions node and the installation/migration node in the test architecture (see Step 5) to populate the data necessary for the other system tests. Use the basic-sanity test(s) as part of the entrance criteria to system testing.

10. Execute and debug each system test developed in Steps 3 through 9. Compare actual results with expected results for each test case in the system test. Investigate each discrepancy found. Based on the user documentation (or equivalent information), determine if the discrepancy is due to a problem in the software, in the documentation of the system under test, or in the test itself. If the problem is with the software or documentation, file a problem report. If the problem is with the system test, fix the test and repeat this step.   [top]

11. For each data variation identified in Step 4.6 that requires an additional execution of the test with different parameters, repeat Step 10 with the data variation.

12. Execute the system tests that have been designed and implemented in the preceding steps over and over again exactly the same way for each successive iteration of the system that reaches the system-test team. Investigate each discrepancy found. Based on the user documentation (or equivalent information), determine if the discrepancy is due to a problem in the software, in the documentation of the system under test, or in the test itself. If the problem is with the software or documentation, file a problem report. If the problem is with the system test, fix the test and repeat the test.

13. Enhance and maintain each system test based on your own experience in executing the test and also based on feedback about the system from users (problems reported). For system defects that reached users but that should have been found by this test, improve the test so that it is capable of finding the defects. Be sure to update the test documentation to cite the specific problem reports that are now covered by the test.

14. After the system-test exit criteria are met and the software has been released to the user organization(s), use measurements collected throughout the process and during production operation to identify lessons learned. Apply these lessons learned to achieve continuous process improvement in the software development life cycle.   [top]

Notes

[1] Test automation and its relationship to the Methodology are discussed in Chapter 18: "System-Testing Tools."

[2] Some authors use the term "data-driven" for parameterized tests and table-driven tests, especially when test automation is involved. With my definition of data dimensionality, making a test data-driven is one approach to achieving this characteristic but it's not the only way.

[3] Risk analysis is discussed in Chapter 13: "Understanding the Typical User."

[4] I first saw the term "Higher-Order Testing" in [Myers, 1979]. Glen Myers used the term "Higher-Order Testing" to refer to all testing beyond module testing (unit testing). Over the years, IÕve found it convenient to reserve the term "Higher-Order System Test" for tests that focus on performance and volume as described in the definition above.

[5] As described in Chapter 6: "The System-Test Oracle," variations exist among industries and companies on how requirements and specifications are prepared, what they are called, and the level of detail in each. The term "requirements/specification process" refers to the overall activity of developing detailed requirements and functional specifications.

[6] Reaching high thresholds is much less of an issue when automated test tools are available, as we discuss in Chapter 16: "The Story of the Test" and Chapter 18: "System-Testing Tools."

[7] Although it can be extremely important to software testing, the capability to change times and dates in an application may need to be surrounded by tight security and management controls in a deployed application. In some cases, changing a time or date in the processes associated with a production application could be bound by legal or regulatory guidelines.

[8] As we discuss in Chapter 16: "The Story of the Test," and in Chapter 18: "System-Testing Tools," the option of parameterizing a test (as an approach to achieving data-dimensionality) becomes more attractive when automated tools are being used.

AddThis Social Bookmark Button

 

 

 


COPYRIGHT NOTICE: This excerpt from System Testing with an Attitude [0-932633-46-3] appears by permission of Dorset House Publishing. Copyright © 2005 by Nathan Petschenik. All rights reserved. See http://www.dorsethouse.com/books/sta.html. The material contained in this file may be shared for noncommercial purposes only, nonexclusively, provided that this Copyright Notice always appears with it. This material may not be combined with advertisements, online or in print, without explicit permission from Dorset House Publishing. For copies of the printed book or for permissions, contact Dorset House Publishing, 1-800-342-6657, 212-620-4053, http://www.dorsethouse.com, info@dorsethouse.com, New: 3143 Broadway, Suite 2B, New York, NY 10027 USA. Additional rights limitations apply, as presented in the Legal Disclaimer posted at http://www.dorsethouse.com/legal.html.

 

 

 
  DORSET HOUSE PUBLISHING CO., INC.
New:3143 Broadway, Suite 2B  New York, New York 10027  USA
1-800-DH-BOOKS or 212-620-4053, fax 212-727-1044
Copyright © 1996-2008 by Dorset House Publishing Co., Inc. All rights reserved.
Home | Blog | Savings | Stores | Features | Titles | Authors | Subjects | Orders | About | Contact | Legal