Software Engineering with Wine - 2010

Contents

Introduction

Popular open source projects like Firefox and Chromium tend to be cross-platform, i.e. they compile and run properly on several operating systems, usually Windows, Mac OS X, and Linux.

This sounds great, but keeping the software building on all three operating systems can be a challenge. Developers working on one operating system can make changes that work great for them, but break the build on other operating systems or compilers.

Automated build-and-test servers help to some extent, but they can get bogged down. It's better if developers can at least compile the app for two operating systems on their own machine before submitting their changes.

However, Linux and Mac developers don't like having to buy a copy of Windows to run it in a virtual machine on their computers. This is where Wine comes in. Wine is a clean-room, open source, and free reimplementation of the Win32 API on top of the Unix API; it can run the Visual C++ commandline compiler on Linux or Mac OS X without a copy of Windows. This makes the build tools for Firefox happy... except for one thing: wine's implementation of cmd.exe doesn't support all the features used by Firefox's build scripts.

In particular, the batch file start-msvc8.bat doesn't work in Wine because Wine's cmd.exe doesn't support the ( and else commands. Likewise, the vcvars.bat batch file that comes with Visual C++ doesn't work.

Worse, Wine doesn't (or didn't until very recently) have a test suite for cmd.exe, so we don't really know what works and what doesn't.

Your mission, should you choose to accept it, is to improve Wine's cmd.exe so it runs vcvars32.bat and start-msvc8.bat properly without breaking anything. (Bonus points for implementing new features like the calculator built in to set /a.)

To succeed, we'll need to tackle the problem from three different angles at the same time.

Task #1: Write Conformance Test Suite for CMD

Wine uses "Test-Driven Development" for its dlls, but not yet for its programs. Even if all we do is get a conformance test for cmd committed to winehq, we'll have made a useful contribution.

The test suite will be a set of batch files and their expected output. They should work identically on the cmd.exe from either Windows or Wine. Here's a prototype of the test suite: cmdtest.tar.gz. The suite consists of the files test-*.cmd and their expected output logs (generated on Windows Vista). There are only two .cmd files at the moment:

You can run those batch files by hand, and compare their output manually, but that gets tedious. cmdtest.sh is a nice way of doing it automatically on your computer. The C program cmd_test.c together with WineTestBot are a nice way of doing it automatically on our server farm running all versions of Windows. That's important because we don't want to commit tests that say that some version of Windows is broken!

Here are some places to get ideas for what tests to write:

  1. all features supported by Wine's current cmd.exe (so any changes we make don't break cmd)
  2. bug reports against wine's implementation of cmd
  3. any other feature used by real cmd batch files in the wild (e.g. Firefox's build scripts or vcvars.bat)
  4. stackoverflow.com's list of favorite cmd features

Task #2: Figure out how the existing parser works and where it is lacking, and prototype the needed improvements

Unix's sh commandline language has to be parsed in several stages, and so does Windows's cmd commandline language. The stages for parsing the cmd language aren't documented anywhere, so wine's existing parser is somewhat incomplete. A post in Raymond Chen's wonderful OldNewThing blog has this to say about cmd:
I don't remember the *exact* steps now, but the first ones are:
  1. join multiple lines (caret escapes newline)
  2. expand %VARIABLES% (caret escapes %)
  3. ...
It gets really horribly painfully complicated from here by the fact that built-in commands and external commands are treated very differently. For starters, they use different tokenizers (that's why "cd.." is parsed as "cd .." and why "cmd.exe" instead stays whole). Then, most built-in commands have their own command-line parser, some changing the syntax considerably

The non-obvious example is "(" - yes, open-parenthesis is a command, yes you can do most "command" stuff with it (included I/O redirection, which unfortunately can get very quirky if you get too creative, like if you try to pipe one "(" into another, and I don't really recommend it), and easily the most powerful one. Its parser digests anything up until an unescaped ")" that occurs outside another built-in command (built-in command parsers nest), and parses multiple lines as multiple statements. It's also the most reliable way to delimit a statement, such as the "true" branch of an IF.

So far, I know of the following shortcomings:

I'm experimenting with a new parser in a tiny prototype parser written in Python. ideas from this should be easy to translate to C if it proves useful.

It may help to use yacc to parse internal commands (wine already eight yacc parsers for various tasks), or then again, maybe the existing builtin command parser is ok. If we do it, we'll probably want to enable it for just one or two builtin commands at first.

Task #3: Incrementally commit improvements to wine's source tree

"Release early, release often" is an important rule of thumb in open source projects. The sooner you get your code out into the real world, the sooner other people can test it and find bugs for you. And code that sits on your hard drive might never see the light of day. Last year's CS130 team implemented dxdiag, but neglected to start the process of sending patches to winehq soon enough. As a result, their code now sits forlorn, waiting for someone to clean it up, break it into small patches, and work to get it committed into the Wine project.

So, to avoid that fate, let's get improved parsing into wine's wcmd incrementally. That means something like this:

  1. contribute the cmd test suite to winehq as it is written
  2. understand the existing code
  3. figure out where to slip in the next improvement. Make sure we have a test case for the improvement.
  4. verify that cmd still passes all tests after the change
  5. submit the change to wine-patches, including new test case (or removing todo_wine from old test case)
  6. address feedback about the change and resubmit as needed

Schedule

Week 1: set up development system, read the existing code

To get you up to speed on the guts of wine's cmd, we will have a kickoff session where we show you how to set up your laptop to develop wine, and walk through how cmd works inside. I'll have a link to a walkthrough here soon.

Week 2: choose a bug or feature, and write and submit a test

There are
many bugs and missing features in wine's cmd, but here is a short list that ought to be easy to get started on: In Week 2, you should choose one of these small issues, write a conformance test that shows the feature working in Windows but not in Wine. Once it passes on your Windows machine (and fails as expected on Linux), and passes on WineTestBot (including the Dutch bot), send it to the project mailing list for review. Once you get an OK there, send it to the wine-patches mailing list.

I'll have a link to a walkthrough for that here soon, too.

Week 3: fix the bug/implement the feature you wrote a test for in week 2

Alternately, if you're working in pairs, you could start on this in week 2 while your partner is working on the test.

Submit your code to wine-patches after it passes review on the team mailing list. (Some features, like set /a, are too big to implement in one week, but you can do at least a trivial implementation for starters.)

Also resubmit your test, with improvements, if it hasn't been accepted yet.

Weeks 4 and 5: keep improving and resubmitting until accepted

If your feature was too big to do in one week, keep working on it.

Also, It's likely that the wine maintainer found some issues with your patch, and hasn't committed it yet. Use this week to figure out what he wants and give it to him. (Remember, the maintainer is always right.)

Week 6: attack problems affecting vcvars32.bat and start-msvc8.bat

Now that you're warmed up, try installing Visual C++ on Wine using winetricks, and try running its vcvars32.bat. (Visual C++ 2005 Express takes about ten minutes to download and install.)

Useful links

cmd reference pages: OldNewThing posts about cmd (sometimes the replies are as interesting as the post): Other:

Project organization

During this project, let's try to stay in close touch.

The main repository is of course the git repository at winehq.org. However, because that one is very challenging to commit to, I'm setting up a git repository at http://github.com/dankegel/wcmd-uplift we can use until our code is ready to submit to winehq.

The mailing list ucla-cs130-wine is set up for students currently enrolled in the class. Please use this to ask questions about the project or run ideas past the coordinator or other students.

I've also set up a chat room for this project using IRC; it's at irc://irc.freenode.net/#ucla-cs130-wine

My favorite way to access IRC is with the Chatzilla plugin to Firefox (Tools / Addons / Get Addons, search for IRC). Then you can just click on the above link, and/or add it to your bookmarks.

Sometimes I use IRCII (sudo apt-get install ircii; irc -c '#ucla-cs130-wine' $USER irc.freenode.net) That's handy when I need to use irc without X (e.g., if my ISP is blocking IRC but not ssh, I can use irc at a remote computer via ssh).

Also, in person weekly meetings might be a good idea. If anyone wants those, let's set them up.


Generic Wine Development Documentation

www.winehq.org has a fair amount of documentation. You should at least have a look at the following:

Setting Up a Wine Development System

  1. Install a fairly recent mainstream linux distribution, e.g. Ubuntu 9.10. (Students with Windows already installed may wish to try the Wubi installer, which installs Ubuntu as if it were a Windows application.)
  2. Install the packages needed to build wine, using the commands
    wget http://winezeug.googlecode.com/svn/trunk/install-wine-deps.sh
    sudo sh install-wine-deps.sh
    
    (see RecommendedPackages).
  3. Get the Wine source tree as described at wiki.winehq.org/GitWine:
    git clone git://source.winehq.org/git/wine.git wine-git
    
    This creates a Wine source tree in a new directory named wine-git.
  4. Build Wine by doing "./configure; make" as usual in the Wine source tree.
  5. Try running some of the programs that come with Wine, e.g.
    cd ~/wine-git
    ./wine programs/cmd/cmd.exe.so
    

Running Wine's Test Suite

Once you have built Wine, it's time to run the conformance test suite. See the instructions at www.winehq.org/docs/winedev-guide/testing-wine. For example, if you're building inside the source tree:
cd ~/wine-git
make -k test  > log 2>&1
(The -k makes it continue past failing tests; the "2>&1 log" redirects both stdout and stderr to a file named "log".) This should take about five minutes. If a test hangs, you may need to kill the process or wineserver from another window.)

To see a list of failures, just look for lines with "Test failed" in them, e.g.

grep "Test failed" log | more
You can get a count of failures with wc:
grep "Test failed" log | wc
You shouldn't see more than ten or so errors in 32 bit Wine (assuming your have an nvidia graphics card).

How To Submit Changes To Wine

See http://wiki.winehq.org/SubmittingPatches.

Once you have made a change to Wine or its test suite that you're happy with, create a patch (see here and here for how), and make sure the patch applies cleanly to a fresh checkout of the Wine sources, and still builds and runs properly. If you used git to retrieve the sources, you should be able to use 'git diff > my.patch' to create a patch.

Once your patch works, add a changelog entry at the top, giving your name and the license (LGPL), and saying very briefly what it does. Then have a friend review the patch for errors, and clean up anything they found confusing or wrong.

If you're a student working in a structured project, send it to your project leader for review, and clean up any issues they find.

Finally, post it to the wine-patches mailing list, and watch for replies on the wine-devel mailing list.

If the patch isn't accepted right away, don't worry, just address any feedback you're given, and resubmit it once a week or so. Ask on wine-devel if you're unsure what's happening. Continue resubmitting until it makes it in. (Sometimes the Wine maintainer's mailbox overflows, so don't expect approval to be quick or easy.)