Honor --python for the Python shebang

waf configure has a stock option --python which controls which Python interpret to use (the "target interpreter"). If unset, the target interpreter is whatever interpreter waf configure runs under.

waf configure uses the target interpreter to determine:

• the default PYTHONDIR / --pythondir and PYTHONARCHDIR / --pythonarchdir
• whether argparse is present
• whether the gps module is present
• whether the ncurses module is present
• the cflags/libs/ldflags for compiling the ntp module (though this will be mooted by !1171 (merged))

The shebang substitutions only honor --pyshebang but not --python. If the user wants to use a non-default Python and sets --python, they can (and usually will) end up with a broken install.

For example, on a Debian or RedHat system with python being Python 2 and python3 being Python 3, if the user runs ./waf configure --python=python3, they are expecting Python 3, but the shebangs will be the default of /usr/bin/env python. This will result in the scripts running under Python 2. That is not what the user expected, nor will it work (because it will fail to find the modules).

If the user makes an explicit request with --python, we should honor that. We can and should allow --pyshebang to override that. This MR implements that behavior.

The Python shebang is detected with the following priority (first match wins):

1. If the user explicitly specifies --pyshebang, use that.
2. If the user explicitly specifies PYTHON or --python, use that. If it is not an absolute path, run under /usr/bin/env.
3. Use /usr/bin/env python as before.

Examples:

These use /usr/bin/env python:

  ./waf configure
  ./waf configure --python=python
  PYTHON=python ./waf configure

These use /usr/bin/env python3:

  PYTHON=python3 ./waf configure
  ./waf configure --python=python3
  PYTHON="/usr/bin/env python3" ./waf configure

These use /usr/bin/python3:

  PYTHON=/usr/bin/python3 ./waf configure
  ./waf configure --python=/usr/bin/python3

---

If we change the default to Python 3 (/usr/bin/env python3), as in !1173, we have the same problem in reverse if the user wants to use Python 2. That is, the user would run ./waf configure --python=python or ./waf configure --python=python2. They are expecting Python 2, but the shebangs will be the default of /usr/bin/env python3. This will result in the scripts running under Python 3. That is not what the user expected, nor will it work (because it will fail to find the modules.)

mentioned in merge request !1166 (closed)

This problem took along time to fix in gpsd. The problem with the MR is that conflates the interpreter used to run waf, with the target interpreter, with the --pyshebang interpreter.

It is common when cross-compileing that the python to run waf is not the python to find the PYTHONARCHDIR and neither is related to the python shebang.

This also gets tabgled up with the hard coded python shebangs in !1173.

I agree that the waf interpreter and the target interpreter are not the same thing. The fact that they are separate is the whole reason that --python exists.

I agree that the shebang and the interpreter are not the same. Specifically:

• /usr/bin/env ...: The shebang runs env; env searches the PATH; env runs the first python it finds.
• Whether found via /usr/bin/env ... or directly used in the shebang as /usr/bin/..., python and/or python3 may be a symlink. e.g. on my system, python -> python2.7 and python3 -> python3.6
• python and/or python3 may not directly be Python at all. For example, on Gentoo, they are (after symlinks) python-exec2c which implements some logic and then exec()s a Python.

It is common when cross-compileing that the python to run waf is not the python to find the PYTHONARCHDIR

Agreed.

and neither is related to the python shebang.

Can you elaborate?

It sounds like this is where we disagree. I think that when the user has specified --python (i.e. a cross Python scenario), the shebang should be derived from the target Python, which is the value of --python. Currently, in the cross scenario, --python has zero effect on --pyshebang, so the shebang defaults to /usr/bin/env python.

"I agree that the waf interpreter and the target interpreter are not the same thing. The fact that they are separate is the whole reason that --python exists."

Good.

"I agree that the shebang and the interpreter are not the same. Specifically:"

Good.

"/usr/bin/env ...: The shebang runs env; env searches the PATH; env runs the first python it finds."

No. I suggest you read "man env". The python it runs need not be, and on some distros is not, in the PATH.

"python and/or python3 may be a symlink."

May be, there also may not be a symlink or file behind those names.

"and neither is related to the python shebang. Can you elaborate?"

When cross-compiling, waf runs a Python on the local host, with PYTHONARCHDIR pointing to python that can only run on the target, and --pyshebang is how python gets run on the target.

Messy yes, but this happens contrantly with gpsd and cross-compilg.

closed

reopened

This is 4 years old. But seems relevant to recent discussions. Is there anything we still need to do here? Or can we close this?

As far as I know, this issue still exists.

However, it will become less important once we drop Python 2 support and make the default some variant of python3. Python 2 vs 3 was a relatively common case of this. I doubt too many people are doing things like specifying a particular python3.x, building against a virtualenv, etc. (Cross-compiling is another case, but in our previous discussions, there was some disagreement and/or lack of clarity on my part as to whether this change improves or hinders cross-compiling. So I'm not going to use cross-compiling as an argument in favor of this change, at least not without researching/testing further, which I have no interest in spending the time to.)

With my Debian hat on: In the Debian package, I use: python3 waf configure --pyshebang=/usr/bin/python3 For Debian Policy (or at least desired practice) reasons, I will always want the direct /usr/bin/python3 not something involving env, so if I have to explicitly specify it there, that's not a big deal.

As a user: If I was a user building from source, I'd be annoyed that with waf configure --python=python3 I made an explicit statement about the desired target Python and then was given something that didn't use that. However, for actual use, I don't build from source. I use the Debian package and everything works.

resolved all threads

closed

assigned to @hal.murray

However, it will become less important once we drop Python 2 support and make the default some variant of python3.

So maybe just wait until python2 is dropped from NTPSec? Maybe this year?

I doubt too many people are doing things like specifying a particular python3.x, building against a virtualenv, etc.

Clearly you are unfamiliar with Gentoo.

As a user: If I was a user building from source,

That is me. Current code works fine for me.

Hal has been looking at this. We;ve been discussing on devel@

I'll assign this to him.