Bug 869

Summary:	suggested test framework enhancements
Product:	ns-3	Reporter:	Tom Henderson <tomh>
Component:	test framework	Assignee:	Craig Dowell <craigdo>
Status:	RESOLVED FIXED
Severity:	enhancement	CC:	craigdo, mathieu.lacage, ns-bugs
Priority:	P5
Version:	pre-release
Hardware:	All
OS:	All

Description Tom Henderson 2010-04-09 18:18:58 UTC

Discussed with Craig today, putting into tracker as reminder

1) would be nice if --basedir=`pwd` were optional parameter; as of right now, must always issue this parameter when debugging in gdb

2) if test fails, test-runner exits with an error code and when trying to run in gdb, there is no stack left around unless an explicit breakpoint set at the point of assertion.  Would be nice if this behaved like other assertions so that gdb could break at that point.

3) if user fails to specify the right name of a test suite, as in --suite=mangled-name, it would be nicer to exit with a "mangled-name test not found" rather than hard error.

Comment 1 Mathieu Lacage 2010-04-13 05:29:13 UTC

it would be nice if --tempdir could default to /tmp too

Also, it would be really nice if we could get rid of the return value of RunTest and rely on the test macros to do our work for us.

Comment 2 Mathieu Lacage 2010-04-13 05:30:26 UTC

also, it would be helpful if --out=test.xml would zero the file if it exists rather than append to it. Right now, it's just confusing when you debug something: you have to remember to always delete the file yourself.

Comment 3 Craig Dowell 2010-04-13 21:54:56 UTC

I just checked in some changes to make life easier for people using test-runner directly.

If tempdir is not provided, test-runner examines the environment variables TMP and TEMP to see if they are set.  If so, it starts with the first of those.  If neither is set, it uses "/tmp".  It then tacks on the current time as a hint to the user which directory is his or hers, and then tacks on a random number.  It then creates a directory and passes that on down to the test suites.  So you get a directory that looks something like,

  /tmp/10.25.57.63889652

for free now.

If basedir is not provided, the test runner finds the current working directory and starts walking up the directory tree looking for a directory file that contains the entries "VERSION" and "LICENSE".  If it finds one, it assumes that must be the base directory and provides it for you.

This means you can now run the test runner directly providing only the suite name as a parameter, as

  ./waf --run "test-runner --suite=pcap-file"

and you can run it under gdb by just doing

  run --suite=pcap-file

I also added a message to be printed if the test-runner can't figure out at least one test to run.  This becomes visible at the test.py level if you do

  ./test.py -v --suite=xxx
  ...
  Unable to find a test to run (constraints too severe or test not found)
  ...

More to come ...

Comment 4 Craig Dowell 2010-04-15 19:42:53 UTC

Added an "--assert" option to test runner.  If you are in gdb, for example, you can;

  run --suite=global-variable --assert

and if an error is detected, the test suite will break (segfault) at the NS_TEST_ASSERT that detected the error.

Comment 5 Tom Henderson 2010-04-16 19:03:26 UTC

Can you add this to test.py?  utils/python-unit-tests.py