Automated Testing Case Study

Background

A Fortune 500 client of ours has invested significantly in the development of a custom Manufacturing Fulfillment system for their 33 worldwide business units. The system is used to manage the production of manufactured products tailored to their customers' needs. The system was originally developed by another software services provider over the course of several years. It is a traditional client/server application developed in Visual C++ and accesses an Oracle database running in a Unix environment.

Our client was receiving weekly updates of the system in the latter phases of development - prior to implementation. They found it virtually impossible to perform the necessary testing and validation of these updates before the next update was received.

They also realized that a more consistent approach was needed to properly test each update of the system.

Objectives

Our client wanted to find a way to perform the testing and validation of these updates to ensure the system was meeting all the functional requirements and was going to be reliable upon deployment. Three Rivers Technologies was asked to help develop a way to automatically test the system quickly and efficiently. The objectives of the project were as follows:

Define a testing strategy that would automate as much testing as possible
Develop Test Plans that would ensure core functionality would be reliably tested using automated testing tools
Develop an automated test suite that would give our client the ability to fully regression test the system quickly during the final phases of development as well as after deployment (as enhancements were being added)
Design and build test scripts for system validation that would be highly reusable and inexpensive to maintain

Methodology/Approach

The first step that Three Rivers Technologies' QA consultants took was to familiarize themselves with the system and its intended functionality. This was accomplished by talking to and working with users of the system, by reviewing requirements documents, and by participating in the existing manual testing activities. The main goal of our early involvement was to gain an overall understanding of the system in order to develop a comprehensive testing strategy that would take maximum advantage of automated testing tools.

Once the overall testing strategy had been established the following tasks were performed:

A Test Plan was developed
A system testing environment was set up that could be restored to its initial state after running scripts - which allowed for repeated execution of the regression tests
Automated test scripts were built using Rational's TeamTest product
Libraries of reusable functions were designed and built so that they could be incorporated into the test scripts in order to minimize maintenance efforts
Automated test scripts that covered 80 percent of the system's functionality were developed over a six month period of time

Results

This project has been very successful. All of the project objectives were met and our client has been extremely happy with the results.

Our client now has the ability to regression test their Manufacturing Fulfillment system in approximately 4 hours using the test scripts our consultants developed. This level of testing would take several weeks to perform manually and would have an increased possibility of missing defects.
Three years after initiating this project, all of the original scripts are still being used to test new releases of the system, and only minimal maintenance has been required to keep scripts updated to work with the new releases.
The automated testing strategy that was envisioned and developed for this system has saved thousands and thousands of hours of manual testing. And because so much of the testing effort has been standardized and automated, testers are able to focus their manual testing efforts for new releases on critical areas of the system that have been enhanced with new functionality.
Automated testing has reduced the time required to deploy new system functionality, thereby increasing users' overall productivity.
The investment made in automated testing of this system has also resulted in improved quality. The system was deployed in the field last year and subsequent updates have been released without any major regression errors being found after release.
The automated test scripts have been leveraged for other types of testing like performance and load testing, environment testing, and integration testing.

Since initiating this project three years ago, our client has gained efficiencies in testing, improved system quality, and realized significant cost savings. As a result, this Fortune 500 client has increased their investment in automated testing and has brought on additional Three Rivers Technologies' QA consultants to design and build automated testing solutions for many of their critical business systems.


	Automated Testing Case Study Background A Fortune 500 client of ours has invested significantly in the development of a custom Manufacturing Fulfillment system for their 33 worldwide business units. The system is used to manage the production of manufactured products tailored to their customers' needs. The system was originally developed by another software services provider over the course of several years. It is a traditional client/server application developed in Visual C++ and accesses an Oracle database running in a Unix environment. Our client was receiving weekly updates of the system in the latter phases of development - prior to implementation. They found it virtually impossible to perform the necessary testing and validation of these updates before the next update was received. They also realized that a more consistent approach was needed to properly test each update of the system. Objectives Our client wanted to find a way to perform the testing and validation of these updates to ensure the system was meeting all the functional requirements and was going to be reliable upon deployment. Three Rivers Technologies was asked to help develop a way to automatically test the system quickly and efficiently. The objectives of the project were as follows: Define a testing strategy that would automate as much testing as possible Develop Test Plans that would ensure core functionality would be reliably tested using automated testing tools Develop an automated test suite that would give our client the ability to fully regression test the system quickly during the final phases of development as well as after deployment (as enhancements were being added) Design and build test scripts for system validation that would be highly reusable and inexpensive to maintain Methodology/Approach The first step that Three Rivers Technologies' QA consultants took was to familiarize themselves with the system and its intended functionality. This was accomplished by talking to and working with users of the system, by reviewing requirements documents, and by participating in the existing manual testing activities. The main goal of our early involvement was to gain an overall understanding of the system in order to develop a comprehensive testing strategy that would take maximum advantage of automated testing tools. Once the overall testing strategy had been established the following tasks were performed: A Test Plan was developed A system testing environment was set up that could be restored to its initial state after running scripts - which allowed for repeated execution of the regression tests Automated test scripts were built using Rational's TeamTest product Libraries of reusable functions were designed and built so that they could be incorporated into the test scripts in order to minimize maintenance efforts Automated test scripts that covered 80 percent of the system's functionality were developed over a six month period of time Results This project has been very successful. All of the project objectives were met and our client has been extremely happy with the results. Our client now has the ability to regression test their Manufacturing Fulfillment system in approximately 4 hours using the test scripts our consultants developed. This level of testing would take several weeks to perform manually and would have an increased possibility of missing defects. Three years after initiating this project, all of the original scripts are still being used to test new releases of the system, and only minimal maintenance has been required to keep scripts updated to work with the new releases. The automated testing strategy that was envisioned and developed for this system has saved thousands and thousands of hours of manual testing. And because so much of the testing effort has been standardized and automated, testers are able to focus their manual testing efforts for new releases on critical areas of the system that have been enhanced with new functionality. Automated testing has reduced the time required to deploy new system functionality, thereby increasing users' overall productivity. The investment made in automated testing of this system has also resulted in improved quality. The system was deployed in the field last year and subsequent updates have been released without any major regression errors being found after release. The automated test scripts have been leveraged for other types of testing like performance and load testing, environment testing, and integration testing. Since initiating this project three years ago, our client has gained efficiencies in testing, improved system quality, and realized significant cost savings. As a result, this Fortune 500 client has increased their investment in automated testing and has brought on additional Three Rivers Technologies' QA consultants to design and build automated testing solutions for many of their critical business systems.
	©2003-2008 Three Rivers Technologies