Bug 449

Summary:	waf regression exit code is zero even if a test fails
Product:	ns-3	Reporter:	Tom Henderson <tomh>
Component:	build system	Assignee:	Gustavo J. A. M. Carneiro <gjcarneiro>
Status:	RESOLVED FIXED
Severity:	normal	CC:	ns-bugs
Priority:	P1
Version:	ns-3-dev
Hardware:	All
OS:	All
Attachments:	patch

Description Tom Henderson 2008-12-17 00:08:27 UTC

Both "./waf --regression" and "./waf --valgrind --regression" return exit code zero even if one of the tests failed.  This makes it harder for scripts such as the nightly regression to check whether there was a failure or not.  

If it is easy to set exit code to one upon a test failure, I would suggest to make that modification; otherwise, I'll have the nightly test look for strings in the log output that indicate errors.

Comment 1 Gustavo J. A. M. Carneiro 2008-12-17 06:26:59 UTC

Created attachment 338 [details]
patch

Sorry about that.  I accidentally broke it recently when refactoring the regression stuff into a separate file (don't remember the bug number).  This patch fixes it.

Comment 2 Tom Henderson 2008-12-18 09:00:13 UTC

(In reply to comment #1)
> Created an attachment (id=338) [details]
> patch
> 
> Sorry about that.  I accidentally broke it recently when refactoring the
> regression stuff into a separate file (don't remember the bug number).  This
> patch fixes it.
> 

That fixed "./waf --regression" (patch was applied) but I still notice a problem with "./waf --valgrind  --regression" which is due to valgrind not reporting an error code.  The below option would fix it, I believe-- I suggest to check it in after the release.

diff -r 2efae18e7379 wscript
--- a/wscript   Wed Dec 17 12:54:16 2008 -0800
+++ b/wscript   Thu Dec 18 05:50:18 2008 -0800
@@ -461,7 +461,7 @@ def get_command_template(*arguments):
     if Params.g_options.valgrind:
         if Params.g_options.command_template:
             Params.fatal("Options --command-template and --valgrind are conflicting")
-        cmd = "valgrind --leak-check=full %s"
+        cmd = "valgrind --leak-check=full --error-exitcode=1 %s"
     else:
         cmd = Params.g_options.command_template or '%s'
     for arg in arguments:

Comment 3 Rajib Bhattacharjea 2008-12-19 13:29:03 UTC

I believe this was fixed for the 3.3 release?  If so, Gustavo, please close it out.

Comment 4 Gustavo J. A. M. Carneiro 2008-12-19 13:59:03 UTC

(In reply to comment #2)
> 
> That fixed "./waf --regression" (patch was applied) but I still notice a
> problem with "./waf --valgrind  --regression" which is due to valgrind not
> reporting an error code.  The below option would fix it, I believe-- I suggest
> to check it in after the release.
> 
> diff -r 2efae18e7379 wscript
> --- a/wscript   Wed Dec 17 12:54:16 2008 -0800
> +++ b/wscript   Thu Dec 18 05:50:18 2008 -0800
> @@ -461,7 +461,7 @@ def get_command_template(*arguments):
>      if Params.g_options.valgrind:
>          if Params.g_options.command_template:
>              Params.fatal("Options --command-template and --valgrind are
> conflicting")
> -        cmd = "valgrind --leak-check=full %s"
> +        cmd = "valgrind --leak-check=full --error-exitcode=1 %s"
>      else:
>          cmd = Params.g_options.command_template or '%s'
>      for arg in arguments:
> 

The patch looks good.  I actually was not aware that valgrind returned 0 even when errors were detected.