Test Automation

Software must be tested to have confidence that it will work as it should in the intended environment. NSPLab^SM has significant experience in the area of test automation and can work with your QA and Engineering organizations to get the tests automated and plugged into your existing automation framework or we can help you setup an automation framework based on our Test-o-matic® automation product.

The rest of this page will take a look at the advantages and disadvantages of manual testing and compare that with automated testing, cover automated unit test, automated GUI testing, and ROI from automating the tests.

Manual Testing

Manual testing requires workers to test software functionality by following a well defined procedure. The problem with manual testing, other than the obvious staffing issues, follow:

• When it comes to testing, human nature is to go with what works. This can lead to manual testers deviating from the well defined procedure so they can "get their job done".
• Manual testers are not always careful in setting up the environment in which the test is run. They believe that action X shouldn't impact the current testing - right?.
• Manual testers are not well suited for performing boring repetitive tasks.
• Computing resources are not always fully utilized depending on manual tester schedules.

Even so it is not always possible, nor desirable to replace all of the manual testing with automated tests. Examples where you may want to keep your manual tests are:

• Tests that are not run often enough to justify the expense of automation.
• Tests where the user interface is very volatile thus creating issues for the automated tests.
• Tests where the results are easily verified by a human but almost impossible to verify via automation.
• Tests that involve physical interaction.

Automated Testing

Test automation can enable some testing tasks to be performed far more efficiently than could ever be done by testing manually. Some of the other benefits are listed below:

• Good regression testing tool provided the tests are easy to run.
• Tie in with code coverage analysis tools to get a better picture of what code has been executed. Code coverage analysis isn't "the solution" for determining the effectiveness of your tests, just one of many tools to determine the effectiveness of your tests.
• Create or tie into existing fault injection tools to get those hard to reach lines of code.
• Run more tests more often increasing your confidence in the product under test.
• Perform tests which would be difficult or impossible to do manually (ex: scalability testing).
• Allows for resources to be more fully utilized.
• Provides repeatable tests.
• Allows for test code reuse through the use of automated test libraries.
• Help your products get to market quicker. Once a set of tests has been automated they can be run in far less time than manual testing would take.

Automated Testing Concerns

There are a number of problems that may be encountered in trying to automate the testing. These concerns follow:

• Test quality. The test cases automated are only as good as the developers that review the test case. Engineering organizations must invest sufficient time in defining and reviewing test cases.
• Maintenance of automated tests. Robustly written automated tests should adapt to product changes. Sometimes product changes may require minor changes to automated test libraries.
• I signed the contract for delivering tests and now my testing priorities have changed. We implement your tests in the order of your priorities making these changes in real time.

Automated Unit Testing

Engineering time is expensive. Why not have the engineer define the unit tests required and then automate them? This will save the engineer time scripting hard coded tests that will rarely be re-used, it gets engineers using your test automation framework and providing invaluable feedback, and adds more test cases to your automation suite.

If you are using code coverage analysis tools then an engineer can pick from the existing automated tests to perform regression testing on the code the engineer is changing.

Automated GUI Testing using Open Source Tools

We have significant experience using PERL-based GUI testing modules that are freely available. Currently these modules are good for basic GUI testing and over time the quality of the modules is constantly improving at no cost to you. We can also add functionality to this module required by your project. We believe this is a good long term approach to GUI testing versus buying expensive tools available from other vendors and being locked into an expensive solution.

Automated Testing ROI

Automation works best when you determine which testing program should be automated to produce the greatest return on investment (ROI). Calculating your ROI can be very difficult. It requires knowing what your current fixed and reoccurring costs are, the cost of automating the tests, and the cost of running and maintaining the automated tests including any new hardware required. In addition, you need to know your QA production now and what you expect it to be after the tests are automated.