Software Engineering with Wine - 2010

Introduction

Popular open source projects like Firefox and Chromium tend to be cross-platform, i.e. they compile and run properly on several operating systems, usually Windows, Mac OS X, and Linux.

This sounds great, but keeping the software building on all three operating systems can be a challenge. Developers working on one operating system can make changes that work great for them, but break the build on other operating systems or compilers.

Automated build-and-test servers help to some extent, but they can get bogged down. It's better if developers can at least compile the app for two operating systems on their own machine before submitting their changes.

This is where Wine comes in. Wine is a clean-room, open source, and free reimplementation of the Win32 API on top of the Unix API; it can run the Visual C++ commandline compiler on Linux or Mac OS X without a copy of Windows. This makes the build tools for Firefox happy... except for one thing: wine's implementation of cmd.exe doesn't support all the features used by Firefox's build scripts.

In particular, the batch file start-msvc8.bat doesn't work in Wine because Wine's cmd.exe doesn't support the ( and else commands. Likewise, the vcvars.bat batch file that comes with Visual C++ doesn't work.

Your mission, should you choose to accept it, is to improve Wine's cmd.exe so it runs those two batch files properly.

But here's the dangerous part: Wine's cmd.exe is not well engineered, and has no test suite. To succeed, we'll need to tackle the problem from three different angles at the same time.

Task #1: Write test Suite for CMD language

The test suite will be a set of batch files and their expected output. They should work identically on the cmd.exe from either Windows or Wine. Many of the tests will be single lines, e.g.
@echo "foo"
which should output
"foo"
(notice the quotes).

Here are some places to get ideas for what tests to write:

  1. all features supported by Wine's current cmd.exe (so any changes we make don't break cmd)
  2. bug reports against wine's implementation of cmd
  3. any feature used by real cmd batch files in the wild (e.g. Firefox's build scripts or vcvars.bat)

We also need a way to mark tests that don't pass on Wine yet, for instance by requiring each test to start with a comment line; if the line contains the string todo_wine, it wouldn't be flagged as a problem if it fails on wine.

Task #2: Figure out how to properly parse the CMD language

The cmd language, like the sh language, has to be parsed in several stages. The stages aren't documented anywhere, but comments on a post in Raymond Chen's wonderful OldNewThing blog touch on it, saying
I reverse-engineered the steps a long long time ago, thanks to a funny quirk in the language. Every step has a distinct set of "special" characters, and for each step the caret only acts as an escape character if followed by one of those, and is ignored otherwise. I don't remember the *exact* steps now, but the first ones are:
  1. join multiple lines (caret escapes newline)
  2. expand %VARIABLES% (caret escapes %)
  3. ...
It gets really horribly painfully complicated from here by the fact that built-in commands and external commands are treated very differently. For starters, they use different tokenizers (that's why "cd.." is parsed as "cd .." and why "cmd.exe" instead stays whole). Then, most built-in commands have their own command-line parser, some changing the syntax considerably

The non-obvious example is "(" - yes, open-parenthesis is a command, yes you can do most "command" stuff with it (included I/O redirection, which unfortunately can get very quirky if you get too creative, like if you try to pipe one "(" into another, and I don't really recommend it), and easily the most powerful one. Its parser digests anything up until an unescaped ")" that occurs outside another built-in command (built-in command parsers nest), and parses multiple lines as multiple statements. It's also the most reliable way to delimit a statement, such as the "true" branch of an IF.

It will probably help to use yacc to parse internal commands (wine already eight yacc parsers for various tasks).

I'm experimenting with a new parser in a tiny prototype parser written in python-lex-yacc; that should be easy to translate to C if it proves useful.

Task #3: Incrementally commit improvements to wine's source tree

"Release early, release often" is an important rule of thumb in open source projects. The sooner you get your code out into the real world, the sooner other people can test it and find bugs for you. And code that sits on your hard drive might never see the light of day. Last year's CS130 team implemented dxdiag, but neglected to start the process of sending patches to wineqa soon enough. As a result, their code now sits forlorn, waiting for someone to clean it up, break it into small patches, and work to get it committed into the Wine project.

So, to avoid that fate, let's get improved parsing into wine's wcmd incrementally. That means

  1. understand the existing code          www.winehq.org has a fair amount of documentation. You should at least have a look at the following:

    Setting Up a Wine Development System

    1. Install a fairly recent mainstream, e.g. Ubuntu 8.10. Students with 64 bit computers should install the 64 bit version of Linux so they have the option of working on the 64 bit port project. (Students with Windows already installed may wish to try the Wubi installer, which installs Ubuntu as if it were a Windows application.)
    2. Install the packages needed to build wine, using the commands
      wget http://winezeug.googlecode.com/svn/trunk/install-wine-deps.sh
      sudo sh install-wine-deps.sh
      
      (see RecommendedPackages).
    3. Get the Wine source tree as described at wiki.winehq.org/GitWine:
      git clone git://source.winehq.org/git/wine.git wine-git
      
      This creates a Wine source tree in a new directory named wine-git.
    4. Build Wine by doing "./configure; make" as usual in the Wine source tree.
    5. Try running some of the programs that come with Wine, e.g.
      cd ~/wine-git
      ./wine programs/cmd/cmd.exe.so
      

    Running Wine's Test Suite

    Once you have built Wine, it's time to run the conformance test suite. See the instructions at www.winehq.org/docs/winedev-guide/testing-wine. For example, if you're building inside the source tree:
    cd ~/wine-git
    make -k test  > log 2>&1
    
    (The -k makes it continue past failing tests; the "2>&1 log" redirects both stdout and stderr to a file named "log".) This should take about five minutes. If a test hangs, you may need to kill the process or wineserver from another window.)

    To see a list of failures, just look for lines with "Test failed" in them, e.g.

    grep "Test failed" log | more
    
    You can get a count of failures with wc:
    grep "Test failed" log | wc
    
    You shouldn't see more than ten or so errors in 32 bit Wine. 64 bit wine has lots and lots and lots of failures still.

    How To Submit Changes To Wine

    Once you have made a change to Wine or its test suite that you're happy with, create a patch (see here and here for how), and make sure the patch applies cleanly to a fresh checkout of the Wine sources, and still builds and runs properly. If you used git to retrieve the sources, you should be able to use 'git diff > my.patch' to create a patch.

    Once your patch works, add a changelog entry at the top, giving your name and the license (LGPL), and saying very briefly what it does. Then have a friend review the patch for errors, and clean up anything they found confusing or wrong.

    If you're a student working in a structured project, send it to your project leader for review, and clean up any issues they find.

    Finally, post it to the wine-patches mailing list, and watch for replies on the wine-devel mailing list.

    If the patch isn't accepted right away, don't worry, just address any feedback you're given, and resubmit it once a week or so. Ask on wine-devel if you're unsure what's happening. Continue resubmitting until it makes it in. (Sometimes the Wine maintainer's mailbox overflows, so don't expect approval to be quick or easy.)

    Mailing list

    The mailing list ucla-cs130-wine is set up for students currently enrolled in the class. Please use this to ask questions about the project or run ideas past the coordinator or other students.

    Chat room

    I've set up a chat room for this project using IRC; it's at irc://irc.freenode.net/#ucla-cs130-wine

    My favorite way to access IRC is with the Chatzilla plugin to Firefox (Tools / Addons / Get Addons, search for IRC). Then you can just click on the above link, and/or add it to your bookmarks.