• Tweet
• Sumo

• Tweet

(creative photo by johnnyberg)

As we’ve discuss in the previous post about testing in Agile software development environment, automation is a very important part to successfully to be agile. However, one doesn’t need to be familiar with Agile process at all to see the immediate benefit of automating the testing process.

This time we’ll see different testing automation tools that we can use to make our life easier. 🙂

Unit Level Testing – xUnit

Unit testing is the very basic level of testing, and it’s important to have unit tests for, if not all, important methods in the application.

Unit testing is specific for different programming language, so you might want to check the xUnit Testing Framework for your project here. For Java, jUnit is the standard.

Code Coverage (White-box Testing)

Unlike the unit testing that is more popular and common, code coverage testing is less known to a lot of developers although they’ve probably heard about white-box testing.

Code coverage testing measures the degree to which the source code is tested. It doesn’t test whether the code produce the output as expected as in the unit testing. It measures whether it is tested or not, and whether all possible path inside it is tested. It also can show whether there are codes that can never get reached (dead code).

Why is this important?

First, it gives a quick picture about how many percentage of the software code is tested. Test engineers and/or developers then can create new test cases or unit test to cover them if necessary. Second, in some industry it is part of industry certification.

The testing can be done in Continuous Integration server, or locally at IDE. You can find the list of tools for your favorite programming language here.

Functional Testing

After all have been said and done, we have to test whether the application is functioning as expected according to the requirements. This testing usually is part that most people called “Application Testing” as it is the most visible one. From my experience, it is also the most time consuming testing in a project and that’s why it can benefitted the most from automation.

There are two types of tools that can be used to perform functional testing:

1. Macro recorders

As the name suggests, the software will record the mouse movement, click, and keyboard keystrokes. The recorded script then can be replayed several times to perform repetitive actions automatically.

In concept it’s similar to keylogger, since both record the user actions, but this one is for a good cause. 🙂

The disadvatage is the window positions must be exactly the same, because it saved the mouse X Y coordinates. If the layout of the application changes, then a new script must be made.

Some of the macro recorder tools are:

• AutoHotKey – free – Windows
• JitBit – commercial – Windows, Mac (beta)
• Xnee – free – X Window

Here is a longer list of macro recorders.

2. Automated testing tool

Fortunately, there is a solution for that X Y coordinates problem, and that is by using a more professional automated testing tool. Previously this type of tool is only accessible if you have deep pockets. Thanks to open-source movement, there are few free alternatives now although it is still dominated by commercial applications.

Here are some of the automated testing you want to check:

1. HP QuickTest Pro

The most popular and considered as the industry standard for automated testing tool, HP QTP provides a complete testing tools (that comes with a hefty price).

Cost: >$5,000
Features:

• Record and playback tests.
• Scripting using VBScript, JavaScript, VC++.
• Support web and non-web applications (VB, VC++, Java, Power Builder, Delphi, .Net, SAP, Siebel, Flash, Oracle Apps, People Soft, Terminal Emulator)
• Support recording on IE only, able to run other browsers (Firefox, Chrome).
• Integration with HP ALM suite for complete development management.

2. Selenium

The most popular open-source automated testing tool, Selenium is unbeatable for the price (free). However, it only support web applications and not as easy as the other commercial tools for non technical users.

Cost: Free (Open-source)
Features:

• Record and playback tests in Firefox. (Selenium IDE)
• Debug and set breakpoints.
• Control web browsers locally or on other computers using programming languages. (Selenium WebDriver)
• Run tests on many servers at the same time. (Selenium Grid)
• Multiple browsers support (record tests on Firefox 3<, run tests on Firefox 3<, IE 7<, Safari, Opera, and Chrome) Open-source extensions (for Flex, Silverlight). Only support web applications. Can be connected with open-source ALM tool such as zAgile.

3. SilkTest (Borland)

Another popular automated testing tool that supports web and non-web applications. Has features that are easy to use by non- technical quality professionals.

Cost: >$4,500
Features:

• Java and .net language scripting support.
• Recorder.
• Debugging through the test steps.
• Support testing for Win32, Web, Java, .net, SAP, RIA (Flex, AIR, Silverlight).
• Multiple browsers support (IE 6, 7, 8, 9, Firefox 3.5 <) Visual analysis. Integration with Borland SilkCentral for traceability matrices.

4. SmartBear TestComplete

Cheaper solution compared to HP QTP and IBM Rational.

Cost: >$1,999
Features:

• Record actions.
• Support testing for .NET, Native Windows applications, Java, Qt, Web applications, HTML5, Flash, Flex, Air, Silverlight.
• Scripting language supported: VBScript, JScript, DelphiScript, C++Script or C#Script.
• OS supported: Windows.
• Browser supported: IE 6-9, Firefox 3.6, 10 – 12, Chrome 19.
• Test reporting and logs.
• Integration with QAComplete for requirement traceability matrices.

5. IBM Rational Functional Tester

Probably second to the HP QTP, IBM Rational Functional Tester is a complete test tools suite for various type of web-based and non web-based applications.

Cost: >$6,240
Features:

• Tool for non-technical quality professionals using visual storyboard.
• Support testing for HTML, Java, Windows, .NET, Visual Basic, Eclipse, SAP, Siebel, Flex, Ajax, Dojo, Silverlight, GEF, PowerBuilder.
• OS supported: Windows, Linux
• Record user actions.
• Eclipse based interface.
• Scripting: Java, VB .NET
• Input from external file.
• Integration with IBM Rational Clearcase.
• Integration with IBM Rational Team Concert for requirement traceability matrices.

6. Seapine QA Wizard

One of the cheaper options for commercial automated testing. Interestingly has its own scripting language to modify the test script.

Cost: >$1,495
Features:

• Also a load-testing tool.
• Support testing for Java, Flash, HTML 5, JavaScript, Silverlight, C#, VB.NET, C++, Win32, Qt, AJAX, ActiveX, and Infragistics Windows Forms controls.
• Browsers supported: IE, Firefox, Chrome.
• Generate test script result reports.
• Proprietary scripting language.
• Integration with Seapine TestTrack for requirement traceability matrices.

What are you waiting for? Go automate your tests!

The benefit of automation is really not obvious, and in fact it might look very time consuming at first. Time is needed to setup the tools, learn how they work, and create the tests themselves.

But once they’re set, the testing that previously take significant time to run manually, are now just few clicks away.

It looks like a very good time investment for me! 🙂

Extra resource:

Selecting an automated testing tool by InfoTech Research Group.