Fixing wine bug #30486

[This page assumes you're already familiar with Linux and Wine, and have read through the resources at wiki.winehq.org/Developers.]

The other day, somebody asked me to help get their app working better in Wine; it seems one little field in one dialog wasn't being drawn. Here's how I went about diagnosing the problem. (I did this while bringing new developer Bill Alley up to speed, and I let him actually write and attach the patch to the bug report.)

Since the app is protected by a dongle, and I didn't have one, I googled around for a demo version. All I could find was an academic version from ten years ago... but I tried it in Wine and was able to reproduce the bug with it. Great!

My first instinct was to look at the stream of Windows messages - maybe Wine was generating them in the wrong order, or leaving one out. Wine has a nice way of logging Windows messages (WINEDEBUG=+message), but I didn't know where to start looking. So I tried Microsoft's Spy++ program. (It's included with Visual C++ 6, and possibly with the downloadable Platform SDK.) This revealed that the dialog field in question had classname msctls_hotkey32, which turns out to be part of comctl32.dll.

And sure enough, installing a native copy of comctl32.dll (by running "winetricks comctl32") made the problem go away! That's not an option for my friend, who wanted Wine to be able to handle his app without any native DLLs, but it does suggest that the bug is in Wine's implementation of the hotkey control.

To do a quick sanity check of Wine's implementation of the hotkey control, I grabbed Raymond Chen's standard skeleton win32 program and added the line

        CreateWindowEx(0, HOTKEY_CLASS, "", WS_CHILD | WS_VISIBLE,
                     15, 10,    // position
                     200, 20,   // size
                     hwnd, NULL, g_hinst, NULL);
right after the existing CreateWindow to add a hotkey control. I then installed mingw32, compiled the modified scratch.c, and ran it with the commands
sudo apt-get install mingw32
i586-mingw32msvc-gcc scratch.c -lcomctl32 -lole32
wine a.exe
Wine's hotkey control displayed perfectly in all its glory, so the problem is more subtle than just a broken control. So I filed wine bug #30486 describing the problem and the workaround. (Probably I should have waited until I'd made more progress, but I wasn't sure I was going to spend any more time on it, and figured it was worth recording what I'd learned.)

At this point, I knew that the control is supposed to be displaying the word "None" initially, and I suspect that there is some problem with windows messages, so I run the app with

WINEDEBUG=+relay,+font,+message wine ledit > log 2>&1
both with and without native comctl32, and look for where the string None shows up in the log. In both cases, the line
trace:font:ExtTextOutW str is L"None"
shows up, so Wine is *trying* to draw the string.

After scratching my head for a while, I look at the region of the log between the ExtTextOutW call and the preceding WM_PAINT message... and notice that in the working case, there is a call to InvalidateRect() before BeginPaint().

So I tried inserting one in Wine's HOTKEY_Paint... and what do you know, now the app draws the hotkey field properly!

Because I don't know if it's a proper fix, I just attached it to the bug report for now.

Total time spent was about six hours, including a few dead ends not recorded above. (A better wine developer could probably have done it a lot faster.)