Improving the Testing Process - A Case Study

By: Michal Tal

How can we ensure the effectiveness of the Testing Process?
How can we be sure we are testing the right thing, investing our resources in the most efficient way to make sure we will detect most of the defects?
The testing manager has to ensure the product’s quality while meeting the timetable and budget, otherwise known as “The Testing Dilemma” (Diagram 1). This is not an easy task. To meet that, the testing process must be efficient and effective.

Measurements can help us collect measurable data about the testing process, asses its effectiveness and improve it.
Some metrics can be extracted on a periodic basis (e.g. weekly) while others can be extracted at the version end. Following are some examples for Defects, development and testing process measurements, and the results analysis and corrective actions, which were implemented during several major releases of a big project in the IT industry.

Defects metrics:

1.    Defect Distribution: SW vendor vs. client
This metric shows the defect detection distribution between the FAT (Factory Acceptance Test) and SAT (Site Acceptance Test) over 3 versions:

• FAT included the Component-Integration Test, done by the developers, and System Test, done by a testing team
•  SAT included the client’s Acceptance Test, done by the customer’s testing team, and Production detected defects.

The SAT defects are considered as “Escaping defects” since they escaped detection during the FAT test levels.

In V1 we had 67% (41%+26%) escaping defects and no defects detected in the Integration Test phase. We took corrective actions in V2:

Project management, Development managers, Developers who performs Integration Test:
•    Allocate more time and resources for Integration Test phase

Project management, Development managers:
•    Monitor the Integration Test planning and execution activities

As a result, the escaping defects were reduced to 51% in V2, and Integration Test defects detection was increased to 12%.
Additional corrective actions were made in V3:

Project management, Development managers, Testing manager, developers who performs Integration Test:
•    Training of test design techniques for developers who preformed Integration Test

Project management, Development managers, Testing manager:
•    Educating development managers with the Integration Test importance

The corrective actions helped. In V3 the escaping defects were reduced to 18% and Integration Test defects detection percentage was increased to 43%.
FAT defect detection was increased to 82% while SAT defect detection was reduced to 18%.

2.    Defects detection distribution to test levels within SW vendor
The following metric shows the defects detection distribution to test levels per week, prior to version delivery to the client.
The black lines are major milestones: Integration Test (Integ. Test) execution start date, System-Test (ST) execution start date and Acceptance Test (AT) execution start date.

As can be seen, only few defects (mostly critical and high severity) are detected in the beginning of Integration Test and ST execution phases.
In addition, there was a sudden increase of defect detection rate during the third week of Integration Test. Further investigation pointed on system instability at that time, caused by changes in the requirements and version scope. This impact the product quality, testing coverage and time table.
The conclusion was that activities must be taken to ensure system stability prior to Integration Test and ST execution phases (such as sanity tests and integration tests review, etc).
Below is a list of the improvements that were implemented:

3.    Defects Root Cause Analysis
The following metric is part of the version’s end metrics. It presents the cause of defects and can indicate the problematic phases in the development process. In order to produce it, the developers have to fill in the necessary information when fixing the defect, in the defect management tool:

Analysis and conclusions:
•    59% of the defects are due to lack of professional knowledge (lack of Programming and technical knowledge, flawed solution (program) design and standards not met) (25%+11%+17%+6%).
    Conclusions: need to improve the programmers’ technical and professional knowledge (for example: by training, on the job coaching, etc).
Development managers, Development Team Leaders
•    Improve the programmers’ technical and professional knowledge by training, on the job coaching, etc
Testing and development managers
•    Coding standards and work procedures training for developers
Development managers, Development Team Leaders
•     Implement code inspection review to verify standards are met and use Code Inspection status report to monitor the progress

•    22% of the defects are due to missing or problematic requirements (15%+7%).
Conclusions: need to take corrective actions to improve the requirements definition process. 
Project management, client, business, development and testing teams
•    Create a Requirements Definition team (client, business, development and testing teams’ representatives) to better define the requirements
•    Create a work procedure and a template for requirement definition
•    Define both functional and non-functional requirements
•    Only signed-off requirements by the client are part of the version’ scope

•    12% of the defects are due to lack of business knowledge.
Conclusions: need to improve the programmers and testers business knowledge (e.g. by learning the client business from the business people, using business processes when applicable, etc).
•    7% of the defects are due to communication problems
Conclusions: need to improve the communication between the different teams in the Project

Development and Testing Process metrics:

1.    Development and Testing Activities – plan vs. actual
The following metric presents the schedule slips in the development and testing activities from the plan:

The green line presents development efforts performed beyond the plan (Development schedule slip), the purple line presents the ST efforts performed beyond AT start (ST schedule slip).
Analysis:
•    Development activities slip of 2 weeks from the plan resulted a slip of 2 weeks in ST start date and a total of 3 weeks slip in the ST activities completion (3 weeks more than planned: planned 6 weeks, actual 9 weeks)
•    The effort on the first week of ST was not used effectively enough since the development was not done yet. That impacted the ST duration and system quality at delivery time.
Conclusions: Need to take actions to identify such risks in advance and handle them in time to prevent such slips
•    Define a Risk Management procedure
•    Define an escalation path and implement it in the project management meetings

2.    Percentage of canceled defects

Canceled defects are not real defects of the system-under-test. They can be the result of environment problem, Requirements out of scope, test design or test execution problem, 3rd party application defect, etc. The percentage of canceled defects may imply about the testing team efficiency and professionalism. Ideally, the canceled defects rate should not be more then 10%.

Analysis:

•    In V1 the canceled defects percentage was 20% of the total defects.
After investigating the root cause of these canceled defects, actions were taken to reduce the canceled defects percentage:

• Training sessions of the system and defect reporting procedure to the testers
• Only experienced testers will do test design
• Implement system walk through with the development teams
• Improve the review process and monitoring

This resulted 7%-11% canceled defects in V2-V5.


3.    Cancel Defects Root Cause Analysis

In order to better understand the causes of the canceled defects, a canceled defects root cause metric was issued:

Analysis:

• 51% (21%+10%+13%+7%) of the canceled defects are due to test design and execution errors and lack of knowledge regarding the system behavior.
  Conclusions: need to improve the testers’ knowledge of the system and test design techniques:
  Testing manager, Development managers and teams
  
  
  
  
  
  
  
  
  1. System training to the testers
  2. Use only experienced testers for the test design
  3. Implement system walk through with the development teams

•  25% (15%+10%) of the canceled defects are due to inappropriate defect reporting procedure (not reproducible and duplicate defects should not be reported according to the procedures of this project).
  
  Conclusions: need to define a work procedure for defect reporting and train the testers accordingly.
• 11% (4%+7%) of the canceled defects are due to Configuration Management problems.
  Conclusion: need to define and implement a better Configuration Management procedure and ensure a better communication between all project’s teams.

Metrics – how to start?

For measurement activities to be cost effective, they must be designed and targeted to support the business goals of the organization and provide effective information for decision making. Specify clearly what results you want to achieve and what behaviors you want to encourage. Metrics should be simple, controllable, and automatable:

• Simple - the popularity, effectiveness, and usefulness of most metrics is inversely proportional to their complexity.
• Controllable - You should tie the success of your testing program to metrics over which you have control. You can control the growth in code coverage and the number of test cases (that is, you can keep adding test code and test cases) but the number of bugs that will be found by the tests is much harder to control.
• Automatable - If calculating a metric requires manual effort it will not be tracked as frequently or as accurately as it should be. It should require little or no human effort to collect the data and calculate the result.

Recommended metrics

Here is a short list of recommended metrics that will help you measure and improve the development and testing process:

• Number of completed test cases/total number of test cases - gives the team a sense for how much work has been completed and how much is remaining.
•  Budget vs. actual effort, cost, and duration: These are standard metrics that point out how the testing team is executing the testing process vs. the original estimates.
• Total testing cost/effort vs. total development cost/effort - provides some perspective on how one project spends its development time vs. other projects.
• Errors by cause: The project team can track and categorize errors that are similar in nature, e.g.:

• Code
• Poor documentation
• Operator error

• Errors by subsystem: If a small number of subsystems have an unusually high number of errors, it can be a sign that more focus needs to be placed in those particular areas.

Recommended books

• Metrics and Models in Software Quality Engineering (2nd Edition), by S. Kan (2002)
• Five Core Metrics: The Intelligence Behind Successful Software Management, by L. Putnam, et al (2003)
• Practical Software Metrics for Project Management and Process Improvement, by R. Grady (1992)
• Measuring the Software Process, by W. Florac (1999)
• Applied Software Measurement: Assuring Productivity and Quality, by C. Jones (1996)
• Practical Software Measurement: Objective Information for Decision Makers by J. McGarry, et al (2001)