Bug 693

Summary:	test framework stops on first error within testcase
Product:	ns-3	Reporter:	Mathieu Lacage <mathieu.lacage>
Component:	core	Assignee:	ns-bugs <ns-bugs>
Status:	RESOLVED FIXED
Severity:	normal	CC:	craigdo
Priority:	P5
Version:	ns-3-dev
Hardware:	All
OS:	All

Description Mathieu Lacage 2009-09-29 06:32:03 UTC

I find this behavior unhelpful: I would rather get information about as many failing asserts as possible than see only the first assert failing within a testcase.

Comment 1 Craig Dowell 2009-09-29 12:06:04 UTC

This is up to the test author based on the macro chosen.

The macros with ASSERT in them will return if an error is detected.  These are used when tests build on the results of previous tests.  For example, if you have a test that ADDs sometthing to a list, and fails, the fact that GETting that same thing from the list will fail is probably quite uninteresting.

The macros with EXPECT in them will continue if an error is detected.  If you have unrelated tests, or need to finish a function, choose these.

I've almost always used the ASSERT flavor in my tests, since the interesting information is generally in the first failure; but you can always use EXPECT.
  
The xml-to-HTML and -text converters will only list the first error, which is probably a bug.

Comment 2 Mathieu Lacage 2009-09-30 07:23:35 UTC

(In reply to comment #1)

> The xml-to-HTML and -text converters will only list the first error, which is
> probably a bug.

I investigated this a bit:

  a) TestSuite::DoRun does not run any testcase within a testsuite beyond the first failing testcase. Easy to fix.

  b) TestCase::Run does not report global testcase failure: it only reports global testcase success (if DoRun does not call ReportFailure but it returns true, the Testcase is reported as neither being successful nor being failing). I believe that this is a user programming error so, we should flag it with an NS_ASSERT. 

  c) The xml format needs to overhauled to support multiple failure reports per testcase (otherwise, the xml to txt/html converters will have a hard time to generate correct output in this case).

Comment 3 Craig Dowell 2009-09-30 19:35:37 UTC

> a) TestSuite::DoRun does not run any testcase within a 
> testsuite beyond the first failing testcase. Easy to fix.

This is obviously a matter of personal preference and situation.  I will make it configurable to report multiple errors or stop at the first.

> b) TestCase::Run does not report global testcase failure:
> it only reports global testcase success (if DoRun does not call 
> ReportFailure but it returns true, the Testcase is reported as 
> neither being successful nor being failing). I believe that this
> is a user programming error so, we should flag it with an
> NS_ASSERT.

Agree.

> c) The xml format needs to overhauled to support multiple failure
> reports per testcase (otherwise, the xml to txt/html converters 
> will have a hard time to generate correct output in this case).

I will make it work in the presence of multiple failures.

Comment 4 Craig Dowell 2009-09-30 23:35:59 UTC

This is now done.  Changeset 0ba73cdd2a43

Use 

  ./test.py -m

or

  ./test.py --multiple

If you want to allow tests to run after errors have been reported.