A short introduction to
DejaGnu

Date

NiklausGiger

Abstract

How to get DejaGnu up and running in a hurry.

1 About this guide

I wrote this instruction, because I (and I don't seem to
be alone) lost a week to figure out how DejaGnu works and
how to write my first test. If I would not be very convinced
that DejaGnu would solve some hard problems for me, I would
have lost patience. Try to follow this instructions as closely
a possible and you should get a good insight. You have been
warned.

I assume that you have no big problems installing DejaGnu
using your package manager or from source. I had no problems
at all with my Debian/GNU/Linux systems (a AMD K6 and a
Mac Powerbook G3) or under Windows NT using the actual cygwin-installer
(Version 1.3.x as of October 2001).

My target system to test is a PPC running vxWorks. As we
do not support more than one target, we do not use gmake
anymore but jam (see http://www.perforce.com). Therefore
I cannot guide you how to use the GNU autotools for cross
compiling. They surely are capable of this.

This example uses the files discussed in chapter 4 "Extending
DejaGnu"

2 Test your installation

Create an empty directory and change the working directory
to it. e.g

ng>: mkdir ~/dejagnu.test

ng>: cd ~/dejagnu.test

Now you are ready to test DejaGnu's main program called ruuntest.
The expecteted output is shown

ng>:~/dejagnu.test$ runtest 

ng>:~/dejagnu.test$ runtest 

WARNING: Couldn't find the global config file. 

WARNING: No tool specified Test 

Run By niklaus on Sun Nov 25 17:07:03 2001 Native configuration
is i586-pc-linux-gnu

=== tests ===

Schedule of variations: unix

Running target unix Using /usr/share/dejagnu/baseboards/unix.exp
as board description file for target. 

Using /usr/share/dejagnu/config/unix.exp as generic interface
file for target. 

ERROR: Couldn't find tool config file for unix.

=== Summary ===

I will show you later how to get rid of all the WARNING-
and ERROR-messages. The files testrun.sum and testrun.log
have been created, which do not interest us at this point.
Let's remove them.

ng>:~/dejagnu.test$ rm testrun.sum testrun.log 

2.1 Windows-NT

You will need a telnet daemon. I didn't actually test this,
as I went strait ahead into using my vxWorks PPC target
to do my remote test. There seems to be a freeware telnet
daemon at http://www.fictional.net/.

Cygwin may be downloaded and installed from a mirror of http://sources.redhat.com/cygwin/.
All examples were also run on Windows NT. I had to download
the sources of DejaGnu to have access to all examples and
the doc. After getting the DejaGnu sources I copies all
files from <dejagnu_sources>/example to /usr/share/doc/dejagnu/example.

3 Create a minimal project, e.g. calc

In this section I will show you how to start a small project,
using the sample application calc, which is part of your
DejaGnu distribution

3.1 A simple project without the GNU autotools

Citing a email by Rob Savoy of Nov 27, 2001 

"The runtest program can be run standalone. All the autoconf/automake
support is just cause those programs are commonly used for
other GNU applications. The key to running runtest standalone
is having the local site.exp file setup correctly, which
automake does."

In our case the generated site.exp had the following content
for our tests.

set tool calc 

set srcdir . 

set objdir /home/Niklaus/dejagnu.test

3.2 Using autoconf/autoheader/automake

We have to prepare some input file in order to run autocon
and automake. There is book "GNU
autoconf, automake and libtool"
by Garry V. Vaughan, et al. NewRider, ISBN 1-57870-190-2
which describes this process thoroughly.

From the calc example distributed with the DejaGnu documentation
you should copy the program file itself (calc.c) and some
additional files, which you might examine a little bit close
to derive their meanings.

ng>:~/dejagnu.test$ cp /usr/share/doc/dejagnu/examples/calc/configure.in .

ng>:~/dejagnu.test$ cp /usr/share/doc/dejagnu/examples/calc/Makefile.am .

ng>:~/dejagnu.test$ cp /data/dejagnu/dejagnu.ng/calc.c .

ng>:~/dejagnu.test$ cp -r /usr/doc/dejagnu/examples/calc/testsuite/ .

In Makemake.am you may not the presence of the AUTOMAKE_OPTIONS
= dejagnu. This option is needed.

Now we run aclocal to generate aclocal.m4, which is a collection
of macros needed by configure.in

ng>:~/dejagnu.test$ aclocal

autoconf is another part of the auto-tools. We have to run
it in order to generate configure based on information contained
in configure.in.

ng>:~/dejagnu.test$ autoconf

autoheader is another part of the auto-tools. We have to
run it in order to generate calc.h.in. 

ng>:~/dejagnu.test$ autoheader

As Makefile.am of this example was developed as port of the
DejaGnu distribution, we have to adapt Makefile.am to our
needs: Replace the line "#check_PROGRAMS
= calc" to "bin_PROGRAMS = calc".
Also the RUNTESTDEFAULTFLAGS require instead of "$$srcdir/testsuite"
a "./testsuite".

Running automake at this point will result in a series of
warning like this:

ng>:~/dejagnu.test$ automake --add-missing

automake: configure.in: installing `./install-sh' 

automake: configure.in: installing `./mkinstalldirs' 

automake: configure.in: installing `./missing' 

automake: Makefile.am: installing `./INSTALL' 

automake: Makefile.am: required file `./NEWS' not found 

automake: Makefile.am: required file `./README' not found 

automake: Makefile.am: installing `./COPYING' 

automake: Makefile.am: required file `./AUTHORS' not found 

automake: Makefile.am: required file `./ChangeLog' not found 

configure.in: 4: required file `./calc.h.in' not found 

Makefile.am:6: required directory ./doc does not exist

Now you might create a empty directory doc and empty (or
putting some meaningfull content into the) files INSTALL,
NEWS, READEM, AUTHORS, ChangeLog and COPYING. The default
COPYING will point to the GNU Public License (GPL). The
GPL preserves us the freedom to distribute, modify and enjoy
useful programs like DejaGnu.

Now it is time to adapt calc to its environment by calling
configu

ng>:~/dejagnu.test$ ./configure 

creating cache ./config.cache 

checking whether to enable maintainer-specific portions of
Makefiles... no 

checking for a BSD compatible install... /usr/bin/install
-c 

checking whether build environment is sane... yes 

checking whether make sets ${MAKE}... yes 

checking for working aclocal... found 

checking for working autoconf... found 

checking for working automake... found 

checking for working autoheader... found 

checking for working makeinfo... found 

checking for gcc... gcc checking whether the C compiler (gcc
) works... yes 

checking whether the C compiler (gcc ) is a cross-compiler...
no 

checking whether we are using GNU C... yes

checking whether gcc accepts -g... yes 

checking for a BSD compatible install... /usr/bin/install
-c 

checking how to run the C preprocessor... gcc -E 

checking for stdlib.h... yes 

checking for strcmp... yes 

updating cache ./config.cache 

creating ./config.status 

creating Makefile creating calc.h

If you are familiar with GNU software, this output should
not contain any surprise to you. And errors should be easy
to fix for such a simple program.

Now make should have no problem to build the calc executable.

ng>:~/dejagnu.test$ make 

gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c 

gcc -g -O2 -o calc calc.o

Not it is time for you to play with calc and verify whether
it work correctly. A sample session might look like this:

ng>:~/dejagnu.test$ ./calc calc: version 

Version: 1.1 

calc: add 3 4 

7 

calc: multiply 3 4 

12 

calc: multiply 2 4 

12 

calc: quit

I'm sure it worked for you too. (Except for the intentional
bug that 2 times 4 equals 12.)

4 Our first automated tests

4.1 Running the test for the calc example

The tests run by DejaGnu need a file called site.exp, which
is automatically generated if we call "make
site.exp". (This was the purpose of the "AUTOMAKE_OPTIONS
= dejagnu" in Makefile.am.)

make site.exp 

ng>:~/dejagnu.test$ make site.exp 

Making a new site.exp file...

Now we are ready to call the automated tests 

ng>:~/dejagnu.test$ make check 

make check-DEJAGNU 

make[1]: Entering directory `/home/Niklaus/dejagnu.test'
srcdir=`cd . && pwd`; export srcdir; \ 

EXPECT=expect; export EXPECT; \ runtest=runtest; \ 

if /bin/sh -c "$runtest --version" > /dev/null 2>&1; then \

$runtest --tool calc CALC=`pwd`/calc --srcdir ./testsuite
; \ 

else echo "WARNING: could not find \`runtest'" 1>&2; :;\ 

fi 

WARNING: Couldn't find the global config file. 

WARNING: Couldn't find tool init file 

Test Run By niklaus on Sun Nov 25 21:42:21 2001 

Native configuration is i586-pc-linux-gnu

       === calc tests ===

Schedule of variations: 

   unix

Running target unix 

Using /usr/share/dejagnu/baseboards/unix.exp as board description
file for target. 

Using /usr/share/dejagnu/config/unix.exp as generic interface
file for target. 

Using ./testsuite/config/unix.exp as tool-and-target-specific
interface file.

Running ./testsuite/calc.test/calc.exp ... 

FAIL: multiply2 (bad match)

=== calc Summary ===

# of expected passes 5 

# of unexpected failures 1 

/home/Niklaus/dejagnu.test/calc version Version: 1.1

make[1]: *** [check-DEJAGNU] Fehler 1 

make[1]: Leaving directory `/home/Niklaus/dejagnu.test' make:
*** [check-am] Fehler 2

Did you the the "FAIL:"
line? The test cases for calc catch the bug in the calc.c
file. If you want you may fix it and rerun the example.

Examine the output files calc.sum and calc.log. Try to understand
the testcases written in /testsuite/calc.test/calc.exp.
To understand Expect you might take a look at the book "Exploring
Expect", which is an excellent resource for learning and
using Expect. (Pub: O'Reilly, ISBN 1-56592-090-2) The book
contains hundreds of examples and also includes a tutorial
on Tcl. Exploring Expect is 602 pages. 

4.2 Getting rid of warnings

DejaGnu may be customized by each user. It first searches
for a file called ~/.dejagnurc. Therefore create this file
and add the following line in it

puts "I am ~/.dejagnurc"

Don't be astonished about the additional line if you call
"make check" again. Next we create ~/site.exp
and add the following line.

puts "I am ~/site.exp"

In a Bash-Shell I entered

ng>:~/dejagnu.test$ export DEJAGNU=~/site.exp

Pointing the global variable "DEJAGNU"
to it and running "make check"
again, kills the "WARNING: Couldn't find
the global config file.". It's time
to create the sub-directory lib and to create the file "calc.exp"
in it, containing one single line, namely

puts "I am lib/calc.exp"

Now, the last "WARNING: Couldn't find tool
init file" has gone. Now we create the directory
~/boards and a file ~/boards/standard.exp. In this file
we add the following line

puts "I am boards/standard.exp"

Now the (abbreviated) output of "make
check" should look like this:

ng>:~/dejagnu.test$ make check 

<...>

fi 

I am ~/.dejagnurc 

I am ~/site.exp 

I am lib/calc.exp 

Test Run By niklaus on Sun Nov 25 22:19:14 2001 

Native configuration is i586-pc-linux-gnu

     === calc tests ===

Using /home/Niklaus/boards/standard.exp as standard board
description file for build. 

I am ~/boards/standard.exp 

Using /home/Niklaus/boards/standard.exp as standard board
description file for host. 

I am ~/boards/standard.exp 

Schedule of variations: 

  unix

Running target unix 

Using /home/Niklaus/boards/standard.exp as standard board
description file for target. 

I am ~/boards/standard.exp 

Using /usr/share/dejagnu/baseboards/unix.exp as board description
file for target. 

<...>

It is up to you to decide when and where you would like to
use one of the above files for customizing. But this chapters
shows you where and in which order the different config
files are run. 

Calling runtest with the '-v'-flag shows you in even more
details which files are searched in which order. Assuming
that the flag $runtest equals runtest the following two
shell commands should produce the exactly same output

make check

runtest --tool calc CALC=`pwd`/calc --srcdir ./testsuite

Now we heard that DejaGnu can run tests on remote hosts.
Let's have a look how this can be done.

4.3 Testing "Hello world"
  locally

This time we will check, whether the built-in shell command
"echo Hello world" will really write "Hello
world" on our console. To test this we create the following
small file "local_echo.exp"
in \~{ }/dejagnu.test/testsuite/calc.test. It should
contain the following lines

set test "Local Hello World" 

send "echo Hello World" 

expect { 

   -re "Hello World"  { pass "$test" } 

}

Run runtest again and verify the output "calc.log"

5 Our first remote test

The easiest remote host is usually the host you are working
on. In this example we will use telnet to login in our own
workstation. For security reason you should never have a
telnet deamon running on machine connected on the internet,
as password and usernames are transmitted in clear text.
I assume you know how to setup your machine for a telnet
daemon.

5.1 Setup telnet to your own host

Try whether you may login in your own host by issuing the
command "telnet 127.0.0.1".
In order to be able to distinguish between my normal session
an a telnet login I added the following lines to my .bashrc.

if [ "$REMOTEHOST" ]

then 

   PS1='remote:\w\$ ' 

fi

Now on my machine a "remote"
login looks like this:

ng>:~/dejagnu.test$ telnet 127.0.0.1 

Trying 127.0.0.1... 

Connected to 127.0.0.1. 

Escape character is '^]'. 

Debian GNU/Linux testing/unstable Linux 

K6Linux login: niklaus 

Password: 

Last login: Sun Nov 25 22:46:34 2001 from localhost on pts/4 

Linux K6Linux 2.4.14 #1 Fre Nov 16 19:28:25 CET 2001 i586
unknown 

No mail. 

remote:~$ exit

logout 

Connection closed by foreign host.

In order to define a correct setup we have to add a line
containing "set target unix"
either to \~{ }/.dejagnurc or to \~{ }/site.exp.
In \~{ }/boards/standard.exp we add the following four
lines to define a few patterns for the DejaGnu telnet login
procedure.

set_board_info shell_prompt    "remote:" 

set_board_info telnet_username "niklaus" 

set_board_info telnet_password "top_secret" 

set_board_info hostname        "localhost"

5.2 Test case: telnet login

The following settings might interfere with DejaGnu pattern
matching of the telnet login process. 

* A non empty/etc/motd

* A non empty/etc/issue.net

* A non existing\~{ }/.hushlogin 

* LANG environment variable is neither empty nor set to "C". 

Now let's try us to setup a test case, which does nothing
more than a login on our "remote"
host. We create the file "remote_echo.exp"
in \~{ }/dejagnu.test/testsuite/calc.test and add the
following few lines:

puts "this is remote_echo.exp target for $target "

target_info $target 

#set verbose 9 

set shell_id [remote_open $target] 

set test "Remote login to $target" 

#set verbose 0

puts "Spawn id for remote shell is $shell_id"

if { $shell_id > 0 } { 

   pass "$test" 

} else { 

   fail "Remote open to $target" 

}

If you have any problems while running runtest, you might 

* out-comment the two lines containing "verbose". 

* setting verbose to a higher number gives you more debug
  information. 

* Pass --debug to the runtest command and examine the new
  output file dbg.log

In the runtest output you should find something like:

Running ./testsuite/calc.test/local_echo.exp ... 

Running ./testsuite/calc.test/remote_echoo.exp ... 

this is remote_echo.exp target is unix 

Spawn id for remote shell is exp7

Have again a look at calc.log to get a feeling how everything
is going on. 

5.3 Remote testing "Hello world"

As a final step we will transform our above "hello
world" example to the remote equivalent. This can be
done by adding the following lines to our file remote_echo.exp.

test "Remote_send Hello World" 

set status [remote_send $target "echo \"Hello World\"\n" ]

pass "$test"

set test "Remote_expect Hello World" 

remote_expect $target 5 { 

   -re "Hello World"  { pass "$test" } 

}

runtest should show you in its output "#
of expected passes 9" and "#
of unexcpected failures 1".

Have a look at the procedures in /usr/share/dejagnu/remote.exp
to have an overview of the possible features. 

5.4 Transferring files from/to the target

A simple procedure like this will do the job for you:

set test "Remote_download" 

set status [remote_download $target \

            /home/niklaus/.dejagnurc \

            /home/niklaus/dejagnu2 ]

puts "status of remote_download ist $status"

The status should return the name of the file on the target. 

??? Here a word of wisdom about the best ways to tranfer
files, load them execute them etc would be very appreciated

5.5 Testing a vxWorks target

In order to test the vxWorks as a target I had to the following
line to standard.exp (and changed the IP, username, password
of course)

set_board_info protocol "telnet"

??? With this setup and some minor modification (e.g. replacing
echo by printf) in my test cases I could test my vxWorks
system. It sure does not seem to be a correct setup by DejaGnu
standard. For instance, it still loading /usr/share/dejagnu/baseboards/unix.exp
instead of vxWorks. In any case I found that (at least under
WindowsNT) I did not find out how the command line would
let me override settings in my personal config files.

5.6 Compiling for the target

??? Here I do not know, how I can accomplish this using the
GNU autotools. Therefore this section will be improved upon
by somebody else.