Bug 2994

Had a brief look at this today; I think the error reported (missing header or library) does not correspond to the actual problem, which is failure to build a python embedded interpreter:

Asking python-config for pyembed '--cflags --libs --ldflags' flags : yes 
Testing pyembed configuration                                      : Could not build a python embedded interpreter

Most likely, something has changed to prevent this from being built successfully, and then pybindgen is not even attempted and the printed error about missing header or library is emitted.

I discovered the following.  Waf detects the presence of python development headers and libraries by trying to compile a python embedded interpreter.  

In bindings/python/wscript, the following is checked:

    try:
        conf.check_python_headers()
    except Errors.ConfigurationError as ex:
        conf.report_optional_feature("python", "Python Bindings", False,
                                     "Python library or headers missing")
        return

This conf.check_python_headers() is defined in waflib/Tools directory, python.py:

def check_python_headers(conf,features='pyembed pyext'):
        features=Utils.to_list(features)
        assert('pyembed'in features)or('pyext'in features),"check_python_headers features must include 'pyembed' and/or 'pyext'"

This code will create a test program (test.cpp; attached) and will try to build it with the attached build script (build-test.sh).

On macOS with Xcode 10, the compilation of this test program (observed in build/config.log) will fail as follows:

/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/unicodeobject.h:534:5: error: 
      'register' storage class specifier is deprecated and incompatible with
      C++17 [-Werror,-Wdeprecated-register]
    register PyObject *obj,     /* Object */

In C++17, the register keyword is not supported.  Apparently, the clang compiler upgrade in Xcode 10 triggers this, despite the presence of -std=c++11

This will lead Waf to conclude that Python headers or development libraries are not found, and pybindgen and pyviz will not be enabled.

Some things that I have tried:

1) disable-werror or -Wno-register.  This leads to a different build error:

Undefined symbols for architecture i386:
  "_Py_Finalize", referenced from:
      _main in test.cpp.1.o
  "_Py_Initialize", referenced from:
      _main in test.cpp.1.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

2) Upgrade waf to the latest (2.0.11) release, assuming that may be worked around by a more portable check_python_headers().

Some possible options:

1) Stop supporting Python 2.7 on macOS.  Is Python3 on macOS an installation hurdle we do not want to impose for ns-3 (it is not default there)?  In general, I suppose we need to start making decisions on maintaining Python 2.7 support.

2) replace the use of the check_python_headers() from waflib/Tools with something else, possibly that we define ourselves.  It could for the time being be limited in scope to macOS.

Created attachment 3191 [details]
build script for test.cpp

Created attachment 3192 [details]
test program that waflib tries (test.cpp)

Moving to GitLab issue 16: https://gitlab.com/nsnam/ns-3-dev/issues/16