Windows GUI test tools and Wine

Windows developers often use a GUI test tool to run system tests of their GUI applications. Ideally, they should be able to run their favorite test tool under Wine.

At the moment, only AutoItV3 and VistaTask, and maybe Win32::GuiTest, seem to work under Wine.

Tools by popularity

In January 2006, I looked around a bit to find out which test tools are popular, using the incredibly silly criterion "number of mentions in comp.lang.*". (Another criterion might be "number of posts in the associated forum at qaforums.com", but I haven't tried that yet.)

(I'm sure I've missed a few important tools. Please email any corrections or suggestions to dank at kegel.com.)

Here are the results:
Name PriceMentions in comp.lang.*
Visual Test Was $600182
winrunner$5000172
AutoIt v3 free 80
win32::GuiTest Free75
Rational Robot$410022
SilkTest$650022
Autotester $300017
Microsoft Scriptit ? 10
CAPBAKn/a 7
Autohotkey free 7
TestComplete$350 4
Vermont High Test$275 3
QuickTest Pro$7500 3
VistaTask $100 0
cxtest Free 0
Eggplant $45000
Tester ThreeFree Hard to search for, but very small
Prices in the above table are from the vendor's web site, from the article "Testing tool makers scramble to keep up" (ADTMag, 10/2001), from versiontracker.com, or from reviews in Indian magazines (translated from Rupees to Dollars :-)

Test results on Wine

I then tested a few with Wine. Here are links to each tool, and results of my tests, if any:

AutoIt3

AutoIt v3 is a freeware BASIC-like scripting language and runtime that can control Windows GUI applications. Wine had trouble running its "remote control notepad" demo, so I filed wine bug 4211, but that turned out to be very minor; Wine's notepad had the wrong window title. So AutoItv3 seems to work under wine-0.9.18 or later. AutoIt used to be open source, but isn't anymore.

ScriptIt

ScriptIt is an old tool mentioned here only because it's where the syntax for AutoIt v2 originally came from.

AutoHotKey

Autohotkey is a GPL'd ScriptIt/AutoIt clone. Some of its function library is copied from AutoIt back when that was open source.

AutoTester

Autotester has not been tested with wine yet, but they're willing to give me an eval copy as soon as I fax them an application.

Cxtest

cxtest is an open source test framework written by Codeweavers that runs on Linux and uses vnc to connect to a system running Windows apps. Currently it is only being used to test apps running on Wine, but potentially it could also be used to test apps running on Windows. Cxtest is currently in use by Codeweavers and a few other people for testing apps running on Wine.

Eggplant

Eggplant is a test framework from Redstone Software that runs on Mac OS/X and uses vnc to connect to a system running Windows or Mac OS apps. It has a free demo, but has not been tested with wine yet (partly because I don't have a Mac to run it on).

QuickTest Pro

I called Mercury to try to get a WinRunner trial, but they pointed me to QuickTest Pro instead. Free trials of QuickTest Pro 6.5 and 8.2 are available. Sadly, neither even installs under wine.

Rational Robot

Rational Robot does not seem to have a free trial. I have not yet tested it under Wine.

TestWorks / CAPBAK/MSW

CAPBAK has a free demo, but has not been tested with wine yet. (The software does not appear to be under active developent, so it's not clear it'd be a good choice.)

SilkTest

SilkTest was by Segue, which has been bought by Borland. There's now an easily downloadable demo (registration required). Sadly, the demo doesn't work under Wine:

TestComplete

TestComplete is a reasonably-priced automated test app recommended by Joel of Joel on Software. Wine currently has trouble with this app:

Tester Three

Tester Three is a free, open-source demo of how to write a GUI testing tool; it was created by John Robbins for a 2002 column in MSDN Magazine. Wine currently has trouble with this app:

Vermont High Test

Vermont High Test is an inexpensive scripting and macro recording test driver. Wine currently has trouble with this app:

VistaTask

VistaTask is an inexpensive general purpose app control tool. It seems to work with wine-0.9.4 -- at least, its canned Notepad demo properly launches Notepad and successfully types text into it. I at first thought it had trouble invoking a two-word menu item under Wine, so I filed wine bug 4221, but it turned out I just had a bug in my script.

I also started a topic on their discussion board:

Visual Test

Visual Test was owned by Microsoft, and then by Rational, and then by IBM, and is no longer sold. There is still a mailing list for Visual Test users. (May have been replaced by Rational Robot.)

WinRunner

winrunner has not been tested with wine yet. (Note: See QuickTest Pro above.) (See also winrunner mailing list)

Perl's Win32::GuiTest

This is tricky to install (you have to like CPAN, and you have to have Visual C++ installed when you install Win32::GuiTest so it can compile the perl extension). Tommy Kho did it, though, and fixed up a few problems in Wine that it tickled. I don't know the current status.

Phantomtest

Installs but doesn't quite work; see

Future ideas

Although it's not really Windows-related, I'd like to mention the Linux Desktop Testing Project. It would be severely cool if the LDTP could be integrated with wine someday; seems like it ought to be possible to write a bridge between AT-SPI (upon which LDTP is based) and Windows' accessibility interface, and run that bridge on Wine. That'd be an interesting project for somebody someday, I bet. (This was discussed once before; see Re: Theming for Wine via the registry, wine-devel, May 2005.)

Links

For more extensive lists of testing tools, see:
Last update: 6 Oct 2006