Windows GUI test tools and Wine
Windows developers often use a GUI test tool
to run system tests of their GUI applications.
Ideally, they should be able to run their favorite test tool under Wine.
At the moment, only AutoItV3 and VistaTask, and maybe Win32::GuiTest,
seem to work under Wine.
Tools by popularity
In January 2006, I looked around a bit to find
out which test tools are popular, using the
incredibly silly criterion "number of mentions in comp.lang.*".
(Another criterion might be "number of posts in the associated
forum at qaforums.com",
but I haven't tried that yet.)
(I'm sure I've missed a few important tools. Please email
any corrections or suggestions to dank at kegel.com.)
Here are the results:
Prices in the above table are from the vendor's web site, from
the article
"Testing tool makers scramble to keep up"
(ADTMag, 10/2001),
from versiontracker.com,
or from reviews in Indian magazines (translated from Rupees to Dollars :-)
Test results on Wine
I then tested a few with Wine. Here are links to each tool,
and results of my tests, if any:
AutoIt3
AutoIt v3 is a freeware BASIC-like
scripting language and runtime that can control Windows GUI applications.
Wine had trouble running its "remote control notepad" demo, so
I filed wine bug 4211,
but that turned out to be very minor; Wine's notepad had the wrong
window title. So AutoItv3 seems to work under wine-0.9.18 or later.
AutoIt used to be open source, but isn't anymore.
ScriptIt
ScriptIt
is an old tool mentioned here only because it's where the syntax for AutoIt v2 originally
came from.
AutoHotKey
Autohotkey is a GPL'd ScriptIt/AutoIt clone.
Some of its function library is copied from AutoIt back when that was open source.
AutoTester
Autotester
has not been tested with wine yet, but they're willing to give me an
eval copy as soon as I fax them an application.
Cxtest
cxtest is an open source test framework
written by Codeweavers that runs on Linux and uses vnc to
connect to a system running Windows apps. Currently it is only
being used to test apps running on Wine, but potentially it could
also be used to test apps running on Windows.
Cxtest is currently in use by Codeweavers and a few other people
for testing apps running on Wine.
Eggplant
Eggplant is a test framework
from Redstone Software that runs on Mac OS/X and uses vnc to
connect to a system running Windows or Mac OS apps.
It has a free demo, but has not been tested with wine yet
(partly because I don't have a Mac to run it on).
QuickTest Pro
I called Mercury to try to get a WinRunner trial, but they pointed me to
QuickTest Pro instead.
Free trials of QuickTest Pro 6.5 and 8.2 are available.
Sadly, neither even installs under wine.
- 4227 QuickTest Pro installer hangs
Rational Robot
Rational Robot
does not seem to have a free trial. I have not yet tested it under Wine.
TestWorks / CAPBAK/MSW
CAPBAK
has a free demo, but has not been tested with wine yet.
(The software does not appear to be under active developent,
so it's not clear it'd be a good choice.)
SilkTest
SilkTest
was by Segue, which has been bought by
Borland. There's now an easily downloadable demo (registration required).
Sadly, the demo doesn't work under Wine:
- 5933 - Borland SilkTest 8.0 trial crashes
TestComplete
TestComplete
is a reasonably-priced automated test app recommended by Joel of Joel on Software.
Wine currently has trouble with this app:
- 4265 - TestComplete painful install, won't start once installed
Tester Three
Tester Three
is a free, open-source demo of how to write a GUI testing tool;
it was created by John Robbins for a 2002 column in MSDN Magazine.
Wine currently has trouble with this app:
- 4209 - tester 3 can't record
Vermont High Test
Vermont High Test is
an inexpensive scripting and macro recording test driver.
Wine currently has trouble with this app:
- 4213 - Vermont High Test can record but playback doesn't work
VistaTask
VistaTask is
an inexpensive general purpose app control tool.
It seems to work with wine-0.9.4 -- at least, its canned Notepad
demo properly launches Notepad and successfully types text into it.
I at first thought it had trouble invoking a two-word menu item under Wine, so I filed wine bug 4221, but it turned out I just had a bug in my script.
I also started a topic on their discussion board:
Visual Test
Visual Test
was owned by Microsoft, and then by Rational, and then by IBM, and
is no longer sold. There is still a mailing list for Visual Test users.
(May have been replaced by Rational Robot.)
WinRunner
winrunner
has not been tested with wine yet. (Note: See QuickTest Pro above.)
(See also winrunner mailing list)
Perl's Win32::GuiTest
This is tricky to install (you have to like CPAN, and you have to have
Visual C++ installed when you install Win32::GuiTest so it can compile the
perl extension).
Tommy Kho did it, though, and fixed up a few problems
in Wine that it tickled. I don't know the current status.
Phantomtest
Installs but doesn't quite work; see
Future ideas
Although it's not really Windows-related, I'd like to mention the
Linux Desktop Testing Project.
It would be severely cool if the LDTP could be integrated
with wine someday; seems like it ought to be possible
to write a bridge between AT-SPI
(upon which LDTP is based) and Windows' accessibility
interface, and run that bridge on Wine. That'd be an
interesting project for somebody someday, I bet.
(This was discussed once before; see Re: Theming for Wine via the registry, wine-devel, May 2005.)
Links
For more extensive lists of testing tools, see:
Last update: 6 Oct 2006