(In reply to comment #0)
> the way header inclusion works for fd-net-device seems to cause bindings
> problems when regenerating on a platform without Planetlab support.
> 
> Rescanning for changeset 3b1c3e88e894 on a machine without planetlab support
> will cause deletion of all PlanetLab-related bindings, and then compilation
> fails for some reason.

I don't manage to reproduce this bug because './waf --apiscan=all' just hangs without doing anything in my machine (probably associated to bug 1622). Could you please provide more information about the bug (exact ./waf command used to scan, host OS where it happened, compilation fail error, etc), this will help me continue debugging. Also would it be possible to get access to the host where this bug was seen?

(In reply to comment #1)
> (In reply to comment #0)
> > the way header inclusion works for fd-net-device seems to cause bindings
> > problems when regenerating on a platform without Planetlab support.
> > 
> > Rescanning for changeset 3b1c3e88e894 on a machine without planetlab support
> > will cause deletion of all PlanetLab-related bindings, and then compilation
> > fails for some reason.
> 
> I don't manage to reproduce this bug because './waf --apiscan=all' just hangs
> without doing anything in my machine (probably associated to bug 1622). Could
> you please provide more information about the bug (exact ./waf command used to
> scan, host OS where it happened, compilation fail error, etc), this will help
> me continue debugging. 

The current bindings work on all machines I've tested.  However, if I rescan them on a machine that does not have PlanetLab enabled:

Tap FdNetDevice               : enabled
Emulation FdNetDevice         : enabled
PlanetLab FdNetDevice         : not enabled (PlanetLab operating system not detected)

I get the following attached patch diff, which basically deletes the planetlab-related bindings (because they are out of the build).

However, with these rescanned bindings, ns-3 will no longer build.  The compilation error is then:

# ./waf build

[ 188/2084] command (${PYTHON}): bindings/python/ns3modulegen-modular.py src/fd-net-device/bindings/modulegen__gcc_LP64.py src/fd-net-device/bindings/modulegen_customizations.py -> build/src/fd-net-device/bindings/ns3module.cc build/src/fd-net-device/bindings/ns3module.h build/src/fd-net-device/bindings/ns3modulegen.log

Build failed
 -> task in 'pybindgen(ns3 module fd-net-device)' failed (exit status 1): 
	{task 33865808: command_task ns3modulegen-modular.py,modulegen__gcc_LP64.py,modulegen_customizations.py -> ns3module.cc,ns3module.h,ns3modulegen.log}


> Also would it be possible to get access to the host
> where this bug was seen?

I will work on this; it is an Ubuntu 10.04-64 machine.

Created attachment 1582 [details]
bindings deleted by rescan on machine without PlanetLab

Created attachment 1587 [details]
Bug fix that forces inclusion of all class headers even if they functionality is not enabled

Created attachment 1588 [details]
Bug fix that ignores missing PlanetLabFdNetDeviceHelper classes when building bindings

(In reply to comment #5)
> Created attachment 1588 [details]
> Bug fix that ignores missing PlanetLabFdNetDeviceHelper classes when building
> bindings

I have attached two patches with two possible solutions. I am not entirely convinced by any of them.

A bit of explanation on why the error occurs:

To scan the bindings waf uses the header files at build/ns3/<modulename>-module.h . There is one such header file per module. If a module is not enabled in a certain machine, no module header file will be created, and waf will not be able to scan the bindings. Additionally, If some classes of a module are not enabled, then they will not be included in the <modulename>-module.h, and they will not be scanned. This is why the PlanetLabFdNetDeviceHelper bindings where deleted when re-scanned in  a machine without Planetlab support.

This is the same behavior present in the TapBridge and Emu modules. Probably bindings are always scanned in a machine that supports both modules, so this problem never appeared before.

The compilation error after rescanning occurs because the presence of the ns3::PlanetLabFdNetDeviceHelper is not being checked in the src/fd-net-device/bindings/modulegen_customizations.py file. This class is not present in root_module.classes if the bindings are not scanned, and so attempting to remove the class from root_module.classes generates an error.

Solutions:

1 - patch #1587: forces inclusion of all classes in the module header even if the class is not enabled. So there will be the headers but the missing classes will not actually be compiled. The benefit of this solution is that all classes will be scanned even if not supported in the current machine. The drawback is that it seems a bit messy to have headers for non supported classes.

2 - patch #1588: everything is as before, so PlanetLabFdNetDeviceHelper bindings will not be scanned , but in the src/fd-net-device/bindings/modulegen_customizations.py the existence of the ns3::PlanetLabFdNetDeviceHelper is validated. This avoids the compilation error when building after re-scanning. The main drawback of this solution is that who wishes to have the PlanetLabFdNetDeviceHelper bindings will need to scan them in a PlanetLab machine.

I am not completely convinced by either solution. So I would like to request advice on which path to follow, or if a better fix can be suggested.

I have not yet tested the patches thoroughly.

(In reply to comment #6)
> (In reply to comment #5)
> > Created attachment 1588 [details]
> > Bug fix that ignores missing PlanetLabFdNetDeviceHelper classes when building
> > bindings
> 
> I have attached two patches with two possible solutions. I am not entirely
> convinced by any of them.
> 
> A bit of explanation on why the error occurs:
> 
> To scan the bindings waf uses the header files at
> build/ns3/<modulename>-module.h . There is one such header file per module. If
> a module is not enabled in a certain machine, no module header file will be
> created, and waf will not be able to scan the bindings. Additionally, If some
> classes of a module are not enabled, then they will not be included in the
> <modulename>-module.h, and they will not be scanned. This is why the
> PlanetLabFdNetDeviceHelper bindings where deleted when re-scanned in  a machine
> without Planetlab support.
> 
> This is the same behavior present in the TapBridge and Emu modules. Probably
> bindings are always scanned in a machine that supports both modules, so this
> problem never appeared before.
> 
> The compilation error after rescanning occurs because the presence of the
> ns3::PlanetLabFdNetDeviceHelper is not being checked in the
> src/fd-net-device/bindings/modulegen_customizations.py file. This class is not
> present in root_module.classes if the bindings are not scanned, and so
> attempting to remove the class from root_module.classes generates an error.
> 
> Solutions:
> 
> 1 - patch #1587: forces inclusion of all classes in the module header even if
> the class is not enabled. So there will be the headers but the missing classes
> will not actually be compiled. The benefit of this solution is that all classes
> will be scanned even if not supported in the current machine. The drawback is
> that it seems a bit messy to have headers for non supported classes.
> 
> 2 - patch #1588: everything is as before, so PlanetLabFdNetDeviceHelper
> bindings will not be scanned , but in the
> src/fd-net-device/bindings/modulegen_customizations.py the existence of the
> ns3::PlanetLabFdNetDeviceHelper is validated. This avoids the compilation error
> when building after re-scanning. The main drawback of this solution is that who
> wishes to have the PlanetLabFdNetDeviceHelper bindings will need to scan them
> in a PlanetLab machine.
> 
> I am not completely convinced by either solution. So I would like to request
> advice on which path to follow, or if a better fix can be suggested.
> 
> I have not yet tested the patches thoroughly.

solution 1 -> attachment 1587 [details]
solution 2 -> attachment 1588 [details]

(In reply to comment #6)
> (In reply to comment #5)
> > Created attachment 1588 [details]
> > Bug fix that ignores missing PlanetLabFdNetDeviceHelper classes when building
> > bindings
> 
> I have attached two patches with two possible solutions. I am not entirely
> convinced by any of them.
> 
> A bit of explanation on why the error occurs:
> 
> To scan the bindings waf uses the header files at
> build/ns3/<modulename>-module.h . There is one such header file per module. If
> a module is not enabled in a certain machine, no module header file will be
> created, and waf will not be able to scan the bindings. Additionally, If some
> classes of a module are not enabled, then they will not be included in the
> <modulename>-module.h, and they will not be scanned. This is why the
> PlanetLabFdNetDeviceHelper bindings where deleted when re-scanned in  a machine
> without Planetlab support.
> 
> This is the same behavior present in the TapBridge and Emu modules. Probably
> bindings are always scanned in a machine that supports both modules, so this
> problem never appeared before.
> 
> The compilation error after rescanning occurs because the presence of the
> ns3::PlanetLabFdNetDeviceHelper is not being checked in the
> src/fd-net-device/bindings/modulegen_customizations.py file. This class is not
> present in root_module.classes if the bindings are not scanned, and so
> attempting to remove the class from root_module.classes generates an error.
> 
> Solutions:
> 
> 1 - patch #1587: forces inclusion of all classes in the module header even if
> the class is not enabled. So there will be the headers but the missing classes
> will not actually be compiled. The benefit of this solution is that all classes
> will be scanned even if not supported in the current machine. The drawback is
> that it seems a bit messy to have headers for non supported classes.
> 
> 2 - patch #1588: everything is as before, so PlanetLabFdNetDeviceHelper
> bindings will not be scanned , but in the
> src/fd-net-device/bindings/modulegen_customizations.py the existence of the
> ns3::PlanetLabFdNetDeviceHelper is validated. This avoids the compilation error
> when building after re-scanning. The main drawback of this solution is that who
> wishes to have the PlanetLabFdNetDeviceHelper bindings will need to scan them
> in a PlanetLab machine.
> 
> I am not completely convinced by either solution. So I would like to request
> advice on which path to follow, or if a better fix can be suggested.
> 

It seems to me that solution 2 is what has been done for other modules elsewhere in the codebase.  I guess that it means that we try to make sure that the main machine that maintains the bindings has all of the optional features enabled when scanning.

(In reply to comment #8)
> (In reply to comment #6)
> > (In reply to comment #5)
> > > Created attachment 1588 [details]
> > > Bug fix that ignores missing PlanetLabFdNetDeviceHelper classes when building
> > > bindings
> > 
> > I have attached two patches with two possible solutions. I am not entirely
> > convinced by any of them.
> > 
> > A bit of explanation on why the error occurs:
> > 
> > To scan the bindings waf uses the header files at
> > build/ns3/<modulename>-module.h . There is one such header file per module. If
> > a module is not enabled in a certain machine, no module header file will be
> > created, and waf will not be able to scan the bindings. Additionally, If some
> > classes of a module are not enabled, then they will not be included in the
> > <modulename>-module.h, and they will not be scanned. This is why the
> > PlanetLabFdNetDeviceHelper bindings where deleted when re-scanned in  a machine
> > without Planetlab support.
> > 
> > This is the same behavior present in the TapBridge and Emu modules. Probably
> > bindings are always scanned in a machine that supports both modules, so this
> > problem never appeared before.
> > 
> > The compilation error after rescanning occurs because the presence of the
> > ns3::PlanetLabFdNetDeviceHelper is not being checked in the
> > src/fd-net-device/bindings/modulegen_customizations.py file. This class is not
> > present in root_module.classes if the bindings are not scanned, and so
> > attempting to remove the class from root_module.classes generates an error.
> > 
> > Solutions:
> > 
> > 1 - patch #1587: forces inclusion of all classes in the module header even if
> > the class is not enabled. So there will be the headers but the missing classes
> > will not actually be compiled. The benefit of this solution is that all classes
> > will be scanned even if not supported in the current machine. The drawback is
> > that it seems a bit messy to have headers for non supported classes.
> > 
> > 2 - patch #1588: everything is as before, so PlanetLabFdNetDeviceHelper
> > bindings will not be scanned , but in the
> > src/fd-net-device/bindings/modulegen_customizations.py the existence of the
> > ns3::PlanetLabFdNetDeviceHelper is validated. This avoids the compilation error
> > when building after re-scanning. The main drawback of this solution is that who
> > wishes to have the PlanetLabFdNetDeviceHelper bindings will need to scan them
> > in a PlanetLab machine.
> > 
> > I am not completely convinced by either solution. So I would like to request
> > advice on which path to follow, or if a better fix can be suggested.
> > 
> 
> It seems to me that solution 2 is what has been done for other modules
> elsewhere in the codebase.  I guess that it means that we try to make sure that
> the main machine that maintains the bindings has all of the optional features
> enabled when scanning.

pushed solution 2. changeset:   211496d6d5d6
please confirm bug is solved to mark as resolved.

the main issue I see now is how to generate these bindings (or keep them from being deleted) from a machine that does not report itself to be of os.type=='onelab'.

One possibility is to add a flag to waf configure that will cause the ENABLE_PLANETLAB to be forcibly set, for the purposes of generating bindings on the reference machine.

Any other ideas?

(In reply to comment #10)
> the main issue I see now is how to generate these bindings (or keep them from
> being deleted) from a machine that does not report itself to be of
> os.type=='onelab'.
> 
> One possibility is to add a flag to waf configure that will cause the
> ENABLE_PLANETLAB to be forcibly set, for the purposes of generating bindings on
> the reference machine.
> 
> Any other ideas?

I don't think this is a good final solution. If the correct expected behavior is that bindings are scanned even if a feature is not enabled, then this should be enforced in the mechanisms the ns-3 waf wscript implement to scan bindings, and not as a workaround in particular modules. If not, what is enforced is that each wscript keeps adding workarounds for particular cases not taken into account in the way ns-3 waf was designed to be used, and eventually things get out of hand. 

However, a refactoring in the way bindings are scanned in ns-3 will certainly take time, and might require modifying not only the main waf binding scanning behavior, but also the wscripts in existing modules. So it is impractical as a solution for now. Maybe this can be considered in the list of requirements for refactoring ns-3 waf wscripts in the future, as was discussed in the last ns-3 community meeting. 

This being said, I don't find another better quick solution than the one you suggested. Anything that forces scanning PlanetLab helper bindings will force to add non supported headers to the build/ns3/fd-net-device-module.h file, so this option might not be something to put as a 'configuration option', but as a 'hidden' environmental variable. In any case the result would be the same.

I can provide a patch in the next couple of days adding the ENABLE_PLANETLAB configuration option/env variable.

(In reply to comment #11)
> (In reply to comment #10)
> > the main issue I see now is how to generate these bindings (or keep them from
> > being deleted) from a machine that does not report itself to be of
> > os.type=='onelab'.
> > 
> > One possibility is to add a flag to waf configure that will cause the
> > ENABLE_PLANETLAB to be forcibly set, for the purposes of generating bindings on
> > the reference machine.
> > 
> > Any other ideas?
> 
> I don't think this is a good final solution. If the correct expected behavior
> is that bindings are scanned even if a feature is not enabled, then this should
> be enforced in the mechanisms the ns-3 waf wscript implement to scan bindings,
> and not as a workaround in particular modules. If not, what is enforced is that
> each wscript keeps adding workarounds for particular cases not taken into
> account in the way ns-3 waf was designed to be used, and eventually things get
> out of hand. 
> 
> However, a refactoring in the way bindings are scanned in ns-3 will certainly
> take time, and might require modifying not only the main waf binding scanning
> behavior, but also the wscripts in existing modules. So it is impractical as a
> solution for now. Maybe this can be considered in the list of requirements for
> refactoring ns-3 waf wscripts in the future, as was discussed in the last ns-3
> community meeting. 
> 
> This being said, I don't find another better quick solution than the one you
> suggested. Anything that forces scanning PlanetLab helper bindings will force
> to add non supported headers to the build/ns3/fd-net-device-module.h file, so
> this option might not be something to put as a 'configuration option', but as a
> 'hidden' environmental variable. In any case the result would be the same.
> 
> I can provide a patch in the next couple of days adding the ENABLE_PLANETLAB
> configuration option/env variable.

Pushed changeset 211496d6d5d6 adding option --force-planetlab . 
PlanetLab helper class will be compiled and bindings scanned, if this option is used, even if the os is not 'onelab'.

(In reply to comment #11)
> (In reply to comment #10)
> > the main issue I see now is how to generate these bindings (or keep them from
> > being deleted) from a machine that does not report itself to be of
> > os.type=='onelab'.
> > 
> > One possibility is to add a flag to waf configure that will cause the
> > ENABLE_PLANETLAB to be forcibly set, for the purposes of generating bindings on
> > the reference machine.
> > 
> > Any other ideas?
> 
> I don't think this is a good final solution. If the correct expected behavior
> is that bindings are scanned even if a feature is not enabled, then this should
> be enforced in the mechanisms the ns-3 waf wscript implement to scan bindings,
> and not as a workaround in particular modules. If not, what is enforced is that
> each wscript keeps adding workarounds for particular cases not taken into
> account in the way ns-3 waf was designed to be used, and eventually things get
> out of hand. 

I agree that the current solution is not ideal, but thanks for doing so for the moment.

Another possible change would be to try to avoid maintaining python bindings for the optional components, and ask users to generate the python bindings themselves.  This requires the bindings generation process to work better, however (the topic of bugs 1622 and 1530).

I'll downgrade the criticality of this now, as well.  The main concern I had was that any user who did an --apiscan=all would have broken their build; this aspect of the problem seems to be now fiexed.

This has now gone several years without additional complaints; suggest to close.

Marking resolved/fixed since Alina's previous commits several years ago have resolved the problem.