Our Blog Excerpts Savings Contact


Dorset House Publishing
High-Quality Books on Software Engineering and Management.  Since 1984.
dorsethouse.com > features

Features       Excerpts       Interviews


iDH Sign-Up

Get Our e-News
Delivered by FeedBurner

The Dorset House Quarterly Interviews

Rodger D. Drabick
Author of Best Practices for the Formal Software Testing Process

ISBN: 978-0-932633-58-3  
©2004  312 pages   softcover  
$35.95 (plus shipping)


DHQ: In your new book, Best Practices for the Formal Software Testing Process, you write that it is the book you wish you'd had at the start of your testing career. Why is that? What errors would you have avoided, knowing what you know now? What's your main message to today's testers?

Drabick: This book is intended to provide a process-related look at the software and system testing life cycle.
     I wrote this book because when I first began software testing, I had previous experience in testing complex hardware systems. Had I just followed that model, I wouldn't have realized the criticality of beginning test planning during requirements definition.
     What errors would I have avoided? See above. Also, I initially didn't realize the importance of test design (i.e., identifying test cases at a high level prior to developing the cases and writing test procedures). Fortunately, I had been introduced to Software Quality Engineering's Systematic Test and Evaluation Process at the very start of my software testing career, as well as being introduced to IEEE Std. 829-1991, the IEEE Standard for Software Test Documentation.
     Many people don't start out with this background. When I began my software testing career in the early 1980Ős, software QA and testing were seen as policemen and adversaries of the development team. Now, we realize that we should be a single team interested in providing products to our customers with a minimum number of defects that satisfy the user's requirements. My main message is that we test engineers and test managers need to optimize our processes and the way we work so that we can help our organizations deliver quality products. I believe we can only do this if we understand our testing process and are continually working to improve it.

DHQ: In his foreword to your book, William Perry describes how your focus changed over the years, until you began concentrating on the testing process. What drew you to the testing process?

Drabick: The answer to this one is simple. After attending many conferences, I realized that process is critical. As a poster on the wall of our work area states, "Action without process is perilous." Such a situation is also chaotic. I had also been exposed to the importance of process by coordinating the first SEI-style self- assessment in Eastman Kodak Company's Research and Engineering Division in 1987. After working on a couple of process definition and improvement initiatives at Eastman Kodak, I realized that understanding a process was the first step to effective process improvement. What solidified my interest in the testing process was the need to create a solid Basis of Estimate for a large program Kodak and IBM were bidding on for the IRS. That's what led to the development of this model. By the way, we won the contract, but that's another story.

DHQ: One especially memorable phrase from the book is that "There is no 'one true way' to test software." What misconception is out there—and where does that one true way mislead testers?

Drabick: Like anything, the way you test depends on the environment you're in. If you're in a DOD, NASA, FDA, or other highly regulated environment, you need a rigorous testing process and approach to testing, because errors can cost people's lives. If you're on a program working a Spiral or other iterative lifecycle, you have need for a defined process for each iteration. In the dot-com environment, you have to develop and test quickly to hit a very limited window of market opportunity. In an XP environment, you're coding and testing together. Each environment demands a different approach to testing. So a testing process model has to be tailorable. One size DOESN'T fit all; one process doesn't apply to all projects. In some environments, an "exploratory testing" strategy may be the best approach, as compared to a more documented, systematic approach.

DHQ: You also assert that testing is not a phase—that testing should start during requirements elucidation and extend through system testing. How is your vision of the scope of testing different from common practice, and what problems does it solve?

Drabick: There are so many visions of the scope of testing that this question is difficult to answer without writing a whole other book. I believe in what I like to call "lifecycle testing." I'd like to hope that with all the work that's been done by good testing consultants and practitioners over the years, my assertion isn't unusual. However, in a number of companies in the commercial world, their development environment is still one where the developers "throw products over the wall" to test engineers, who in some cases don't even know there's a new product being worked on. (I worked at a small company like this a few years ago. Surprising the testers with a delivery doesn't say much for management, the test engineer, or the development engineer's ability to communicate, but that's another issue.) If test engineers have a customer's interests and perspective in mind, they can ask significant questions during requirements analysis. Test engineers should also be planning their tests at that time, so that when coding and debugging is complete, the test team has its test procedures ready to run and the test environment in place to properly evaluate the quality of the product. But let there be no mistake; the developers are responsible for the quality of the product. Test and QA engineers measure the quality of the product, but you don't achieve zero-defect code through the application of testing alone.
     Note that I consider "testing" and "SQA" to be two different disciplines. This philosophy came from the fact that IEEE has separate standards for SQA plans (IEEE-Std-728) and Test plans (IEEE-Std-629). I happen to have a similar process model for SQA, if anyone's interested.

DHQ: You are a strong advocate of formal test plans. How should testers develop a test plan? What's the key to developing an effective test plan?

Drabick: I describe how to develop a test plan in Chapter 4 of my book, and it is also described in books by Bill Perry, Rex Black, Rick Craig, and others. There are several keys to developing an effective test plan, some of which are described in Chapter 4 of my book (Create Test Plan). They involve creating a workable test strategy for your project and environment, synchronizing your test plan with the other program plans (see Chapter 3 of my book, Review Program Plans), and in having a testable set of requirements (see the section in Chapter 4 on Reviewing Requirements).
     Test plans don't have to pass a weight or page-count test. If you can write a test plan for your environment that adequately defines your test strategy and how you'll test the functionality in your system in 5 pages or less, great. Most of the time it takes a longer document, but the key is to use what works for you. I personally favor a test plan written according to the format of IEEE-Std-829, but again, use what works and what your management will tolerate.

DHQ: In the book, you discuss ways that the formal testing process can be applied in an Extreme Programming (XP) environment, in which the development schedule is highly compressed. Is the formal testing process compatible with XP? How might XP suffer from omitting parts of the process?

Drabick: For the answer to this question, you'll have to read Chapter 8 of my book (Tailoring the Model).
     Some XP projects won't have a formal testing group, in which case they won't need a formal test process. However, they should define the test process they are using, which will normally be a form of "buddy testing," per Rick Craig. In XP, there isn't time for a full test planning/test design/test case/test procedure cycle with each cycle of XP development. A single test plan or strategy should be developed at the start of the XP program. Then, develop your automated test scripts to support each release cycle, keeping in mind the test strategy you identified early on. If the project changes scope and direction as it matures, you might consider updating the strategy. For an excellent treatise on this subject, see Lisa Crispin and Tip House's book, Testing Extreme Programming.

DHQ: What would you say is the biggest problem in testing today, and how can it be remedied?

Drabick: Wow, that's a tough question, and one that I'm sure a variety of people would answer in different ways. My answer is, our biggest problem is management that doesn't understand the importance and the significance of testing. I just had a discussion on this subject today with a test engineering colleague. A sharp test engineering team is too often undervalued and unappreciated by management. Little in the way of resources or training is provided to the test team, and then it is soundly criticized when poor products are released (often after cutting the time allowed to perform testing, not providing an adequate test environment, and not bringing the test team on-board when requirements were being developed). Management conveniently forgets that the test team didn't build the failure-prone software. That's another reason I wrote my book, so that management could get a quick appreciation for what testing is all about.

DHQ: What can we do to help testers and developers see themselves as a part of the same team, rather than members of different departments? How can they work more cooperatively?

Drabick: Break down "the walls" between groups; make sure your test team is co-located with your developers. I remember a big project where my test team was on the seventh floor of a building, and the developers were on the second floor. There was only a single stairway and a slow freight elevator between the floors. Additionally, the test lab was in the basement. Communication was rotten on that project. Staff your test team early, and get the test team involved in the project during requirements development. Encourage developers and the test team to share lunch-time learning sessions, where the developers talk about their development techniques, and the test team makes presentations about their test process, their test techniques, and new things they are learning and doing. And have management say good things and express appreciation for the efforts of both groups, in staff meetings. At the end of the project, hold a project retrospective using the techniques outlined in Norman Kerth's book with all members of the cross-functional project team.

DHQ: What do you say to those who interpret the words "formal" and "process" as "rigid" and "slow"?

Drabick: As I state relatively early in my book, everybody has a process. It's "the way you do things." The question is, should you document that process? The whole point about my process model, especially as pointed out in Chapter 8, Tailoring the Process, is that you need to set up the process that works for you, and continually tune it based on quantitative metrics. That removes the "rigid" objection. One significant point of the Capability Maturity Model – Software, and the newer Capability Maturity Model – Integration is that your process must be continually improved based on quantitative metrics. If you have a process that's cast in concrete, it will be ignored. Again, as was mentioned above, the process you use has to depend on your specific environment.

DHQ: What companies and/or types of environments are implementing the formal software testing process you describe? Where can we find more information about their experiences?

Drabick: As mentioned earlier, every organization has a testing process, whether it realizes it or not. The question is, how formal and well documented is it? Both ISO and CMM/CMMI push organizations in this direction. This would indicate that many large and medium-sized companies in the more typical, non-Web lines of business have a formal testing group. Smaller and Web-based companies may not have a defined process, but that doesn't mean they don't need one. I've used the process in a division at the Eastman Kodak Company, I used it in a small company in the medical document processing industry, and I'm working to get my current company to adopt parts of it. To find out information, go to software QA and testing conferences, like the International Software Testing Conference, hosted by the Quality Assurance Institute; the STAR Conferences, hosted by Software Quality Engineering; and Quality Week, sponsored by Software Research, Inc. I'm sure there are others. You could also consult the QAI and SQE Websites; again, I'm sure there are a variety of other places to look on the Web.

DHQ: What prevents testers and managers from defining a formal testing process? How can an individual reader of this book influence the implementation of the process you describe?

Drabick: What prevents an organization from defining a formal testing process is the fact that until now, the definition required a lot of work, and it costs money. My book and the embedded process model should ease this effort. Most companies working in highly regulated industries have to have such a process, or they won't be awarded contracts. Many subcontract development shops have to have a formal testing group (though their process may not be defined), or they won't be awarded contracts either. Qualifying for a subcontract may require a CMM/CMMI rating of 3 or higher. Such a rating needs a defined testing process.
     Readers of this book can take some off-work time to download the PowerPoint or PDF charts for the model from the Dorset House Website (www.dorsethouse.com/books/bpf.html), and then mark up those charts to match their current process. After that, if the reader is not a manager, he or she could approach his or her test manager with a proposal to document the testing process. After that, it's a matter of deciding how to improve that process using an ISO or CMMI approach.

DHQ: Why should experienced testers, test managers, team leaders, or new testers read this book? Who else should read it?

Drabick: Experienced test engineers, test managers, and team leaders should read this book if they are interested in defining and documenting their existing testing process as the first step to improving it. The diagrams in this book can also be used to teach development personnel and managers what is involved in the testing process. There's a lot more to it than they might expect.
      New testers will find a definition of the overall testing process in this book, so it will provide them with a broader perspective on testing.
      Other team leaders and managers in the development organization will be able to understand all that is involved in the testing process. If you're a CEO or CFO of a small company looking to develop high quality products, and you already have a testing group in place or are considering implementing such a group, this is the book for you.
      I'd also highly recommend this book to computer science or software engineering students. There aren't many testing courses out there yet, and everyone who's going to be developing software should know how a testing process works. Who knows, you might even decide to pursue a career as a test engineer.

DHQ: What kinds of college courses should consider this book for adoption? Where would this book fit into an undergraduate or graduate software engineering curriculum?

Drabick: I believe that this book could provide the basis for an Introductory level bachelor's quarter or semester course in software and system testing. Homework and project assignments could be developed (linked to a programming Case Study) in requirements review, developing test plans, developing test design, developing test cases, developing test procedures, and then executing the tests. After the tests have been executed, students could exchange the final tested product with other students, who would represent users, to see if they can find additional defects that had managed to escape detection. Since developers also understand what a formal testing group is doing, this book would provide a basis for a supplement to a sophomore-level course in software engineering or computer science.

DHQ: Now that your first book has been published, what's next? What kind of work are you doing these days?

Drabick: Actually, I'm not doing testing these days; I'm the lead SQA engineer on the Canadian Census project at Lockheed Martin's Transportation and Security Solutions Business Group. We're preparing a system to assist the Canadian government in their 2006 census. It's an exciting project, and I'm working with an excellent team. We're in the middle of our formal testing cycle right now, which means I have the opportunity to assist our test team by reviewing their test plans and test procedures prior to formal inspections; I also witness the formal tests.
     Though not in the near future, there may be another book in the works. Some years ago I wrote an article for Software QA Magazine titled "Ethics in Software QA and Testing." A publisher approached me about writing a book on that subject, but I wasn't ready to write that book then. Given the state of industry these days, I think that might be an interesting book to write.

DHQ: So what does a reader do after reading your book?

Drabick: I'm going to steal a page from a number of post-conference presentations Bill Perry used to make. What you should do, is select two or three things that you want to do immediately with what you have learned, and begin working on those items.
     What you select will depend on who you are and where you work. For example, if you are a junior test engineer in a company at CMMI Level 3 or above, you should ask your testing lead or manager to describe the formal testing process to you. You might ask him or her for any documentation available on that process. If the process isn't documented, you might mention this book and suggest that your group begin documenting your testing process. That will help management and the other cross-functional groups in your project understand just what it is your test team is doing.
     As another example, if you're a testing manager in a small company where software gets "thrown over the wall," you should quickly download the Testing Process Model charts from the Dorset House Website (www.dorsethouse.com/books/bpf.html), and document your current process. Then, you might use the information in Chapter 4 to begin analyzing requirements for testability. You might also begin a process of formally reviewing/inspecting your test plans/test designs/test procedures with the development team. That way, both your test group and your development group will know what you are planning to test, so that any disconnects can be identified.
     There are a number of other things you could do, depending on who you are and the environment you work in. Hopefully, these examples will give you some pointers as to things to consider.

AddThis Social Bookmark Button





COPYRIGHT NOTICE: The material contained in this file may be copied or distributed freely, provided that the material is copied or distributed in its entirety, including this Copyright Notice. This material is Copyright © 2002 by Dorset House Publishing Co., Inc. No use may be made of the material without acknowledgment of its source: Dorset House Publishing Co., http://www.dorsethouse.com, info@dorsethouse.com, New: 3143 Broadway, Suite 2B, New York, NY 10027 USA. Additional rights limitations apply, as presented in the Legal Disclaimer posted at http://www.dorsethouse.com/legal.html.






New:3143 Broadway, Suite 2B  New York, New York 10027  USA
1-800-DH-BOOKS or 212-620-4053, fax 212-727-1044

Copyright © 1996-2008 by Dorset House Publishing Co., Inc. All rights reserved.
Home | Blog | Savings | Stores | Features | Titles | Authors | Subjects | Orders | About | Contact | Legal