What is new in this release ?

This release has a number of substantial changes over version 1.2. The most visible change is that the version of expect and Tcl included in the release are up-to-date with the current stable net releases. Other changes are:

1. The config sub-system in DejaGnu has been completely redesigned. It now supports testing on remote hosts as well as remote targets.
2. More builtin support for building target binaries with the correct linker flags. Currently this only works with GCC, preferably with a target support by libgloss.
3. Lots of little bug fixes from a year of heavy use here at Cygnus Support.
4. DejaGnu now uses autoconf for configuration.
5. New test cases for DejaGnu have been added for the new features, plus the "--tool" option bug in the 1.2 testsuite has been fixed.
6. The --tool option is now optional.
7. runtest when searching for test drivers ignores all directories named SCCS, RCS, and CVS.
8. There is now a generic keyword based test harness that uses comments in source code to control how each test case gets built and run.
9. There is now some support for running a testsuite with multiple passes.

Running existing tests

To run tests from an existing collection, first use configure as usual to set up the source directory containing the tests. Then try running

make check

If the check target exists, it usually saves you some trouble--for instance, it can set up any auxiliary programs or other files needed by the tests.

Once you have run `make check' to build any auxiliary files, you might want to call the test driver runtest directly to repeat the tests. You may also have to call runtest directly for test collections with no check target in the `Makefile'.

Typically, you must use two command-line options: `--tool', to specify which set of tests to run(1), and `--srcdir', to specify where to find test directories.

For example, if the directory `gdb/testsuite' contains a collection of DejaGnu tests for GDB, you can run them like this:

eg$ cd gdb/testsuite
eg$ runtest --tool gdb
Test output follows, ending with:

		=== gdb Summary ===

# of expected passes 508
# of expected failures 103
/usr/latest/bin/gdb version 4.14.4 -nx

You can use the option `--srcdir' to point to some other directory containing a collection of tests:

eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite

These examples assume a native configuration, where the same computer runs both runtest and the tests themselves. When you have a cross configuration, the tests run on a different computer, controlled by the host running runtest. In this situation, you need the option `--name' to specify the network address for the other computer:

eg$ runtest --tool gdb --name vx9.munist.com

If you always use the same option values, you can record them in a file called `site.exp', rather than typing them each time. See section Config Variables.

By default, runtest prints only the names of the tests it runs, output from any tests that have unexpected results, and a summary showing how many tests passed and how many failed. To display output from all tests (whether or not they behave as expected), use the `--all' option. For more verbose output about processes being run, communication, and so on, use `--verbose'. To see even more output, use multiple `--verbose' options. See section Using runtest, for a more detailed explanation of each runtest option.

Test output goes into two files in your current directory: summary output in `tool.sum', and detailed output in `tool.log'. (tool refers to the collection of tests; for example, after a run with `--tool gdb', look for output files `gdb.sum' and `gdb.log'.) See section The files DejaGnu writes.

What does a DejaGnu test look like?

Each DejaGnu test is an expect script; the tests vary widely in complexity, depending on the nature of the tool and the feature tested.

Here is a very simple GDB test--one of the simplest tests shipped with DejaGnu (extracted from `gdb.t00/echo.exp'):(2)

# send a string to the GDB stdin:
send "echo Hello world!\n"

# inspect the GDB stdout for the correct reply,
# and determine whether the test passes or fails:
expect {
  -re "Hello world.*$prompt $"    { pass "Echo test" }
  -re "$prompt $"                 { fail "Echo test" }
  timeout                         { fail "(timeout) Echo test" }
  }

Though brief, this example is a complete test. It illustrates some of the main features of DejaGnu test scripts:

• The test case does not start the tested program (GDB in this case); all test scripts for interactive tools can assume the corresponding tool is running.
• Comments start with `#'.
• The main commands you use to control a tested program are send (to give it commands) and expect (to analyze its responses).
• The expect command uses a list of pairs; a pattern (regular expression if `-re' specified), followed by an action to run if the pattern matches output from the program. Only the action for the first matching pattern will execute.
• Test cases use the commands pass and fail to record the test outcome.

Design goals

DejaGnu grew out of the internal needs of Cygnus Support. Cygnus maintains and enhances a variety of free programs in many different environments, and we needed a testing tool that:

• is useful to developers while fixing bugs;
• automates running many tests during a software release process;
• is portable among a variety of host computers;
• supports cross-development testing;
• permits testing interactive programs, like GDB; and
• permits testing batch oriented programs, like GCC.

Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to make sure that GDB works as well when cross-debugging as it does in a native configuration.

Probably the greatest challenge was testing in a cross-development environment (which can be a real nightmare). Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communication interfaces vary from a serial line to ethernet. DejaGnu was designed with a modular communication setup, so that each kind of communication can be added as required, and supported thereafter. Once a communication procedure is coded, any test can use it. Currently DejaGnu can use rsh, rlogin, telnet, tip, kermit, and mondfe for remote communications.

Julia Menapace first coined the term "Deja Gnu" to describe an earlier testing framework at Cygnus Support. When we replaced it with the Expect-based framework, it was like DejaGnu all over again...

A POSIX conforming test framework

DejaGnu conforms to the POSIX standard for test frameworks.

POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET.(3)

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. DejaGnu supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

PASS

A test has succeeded. That is, it demonstrated that the assertion is true.

XFAIL

POSIX 1003.3 does not incorporate the notion of expected failures, so PASS, instead of XPASS, must also be returned for test cases which were expected to fail and did not. This means that PASS is in some sense more ambiguous than if XPASS is also used. For information on XPASS and XFAIL, see section Using runtest.

FAIL

A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework. As with PASS, POSIX tests must return FAIL rather than XFAIL even if a failure was expected.

UNRESOLVED

A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when writing tests, to not carelessly use tcl statements like return---if you alter the flow of control of the tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up UNRESOLVED:

• A test's execution is interrupted.
• A test does not produce a clear result. This is usually because there was an ERROR from DejaGnu while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened--and to improve the test case.
• A test depends on a previous test, which fails.
• The test was set up incorrectly.

UNTESTED

A test was not run. This is a placeholder, used when there is no real test case yet.

The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

UNSUPPORTED

There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. DejaGnu also uses this message when a testing environment (often a "bare board" target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor.

DejaGnu uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, you must avoid the setup_xfail procedure as described in the PASS section above, and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above.

Future directions

In the near future, there are two parallel directions for DejaGnu development. The first is to add support for more hosts and targets.

The second would permit testing programs with a more complex interface, whether text based or GUI based. Two components already exist: a Tcl based X window toolkit, and a terminal package for expect. Both of these could be merged into DejaGnu in a way that permits testing programs that run in each environment.

Meanwhile, we hope DejaGnu enables the creation of test suites for conformance to ANSI C and C++, to POSIX, and to other standards. We encourage you to make any test suites you create freely available, under the same terms as DejaGnu itself.

Tcl and Expect

Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter Usenix conference, Tcl: An Embeddable Command Language. That paper is included in PostScript form in the `doc' subdirectory of the Tcl distribution. The version of Tcl included in DejaGnu at this time is Tcl 7.4p3.

Don Libes introduced expect in his paper expect: Curing Those Uncontrollable Fits of Interaction at the 1990 Summer Usenix conference. The paper is included in PostScript form in the expect distribution (as are several other papers about expect). The version of expect included in DejaGnu at this time is expect 5.18.0.

Go to the first, previous, next, last section, table of contents.