Dear Brother Yocto,

Here’s a weird bug I found on my setup: pythonnative seems to be using python3.4.3 whereas the recipe specified 2.7.

What I tested

I found the code in phosphor-settings-manager, which inherits pythonnative, seems to be working per python 3, so added the following line into a native python function:

python do_check_version () {
import sys
  bb.warn(sys.version)
}
addtask check_version after do_configure before do_compile

WARNING: phosphor-settings-manager-1.0-r1 do_check_version: 3.4.3 (default,
Nov 17 2016, 01:08:31)

This is with openbmc git tip, building MACHINE=zaius

What the recipe should be using is seemingly 2.7

$ cat ../import-layers/yocto-poky/meta/classes/pythonnative.bbclass
inherit python-dir
[...]
$ cat ../import-layers/yocto-poky/meta/classes/python-dir.bbclass
PYTHON_BASEVERSION = "2.7"
PYTHON_ABI = ""
PYTHON_DIR = "python${PYTHON_BASEVERSION}"
PYTHON_PN = "python"
PYTHON_SITEPACKAGES_DIR = "${libdir}/${PYTHON_DIR}/site-packages"

Why this is bad

well, python2.7 and python3 have a ton of differences. A bug I discovered is due to usage of filter(), which returns different objects between Python2/3.

Before diving into the layers of bitbake recipes to root cause what caused this, I want to know if you have ideas why this went wrong.

—Snakes in a Sysroot

Dear Snakes in a Sysroot,

As you’ve already discovered, the version of Python used varies depending on context within a Yocto build. Recipes, by default, are built to run on the target machine and distro. For most recipes, that is what we want: we’re building a recipe so it can be included in the rootfs. In some cases, a tool needs to be run on the machine running bitbake. An obvious example is a compiler that generates binaries for the target machine. Of course, that compiler likely has dependencies on libraries that must be generated for the machine that runs that compiler. This leads to three categories of recipes: native, cross, and target. Each of these categories place their output into different directories (sysroots) both to avoid file collisions and to allow BitBake to select which tools are available to a recipe by altering the PATH environment variable.

native

Binaries and libraries generated by these recipes are executed on the machine running bitbake (host machine). Any binary that generates other binaries (compiler, linker, etc) will also generate binaries for the host machine.

cross

Similar to native, the binaries and libraries generated by these recipes are executed on the host machine. The difference is that any binary that generates other binaries will generate binaries for the target machine.

target

Binaries and libraries generated by these recipes are executed on the target machine only.

For a recipe like Python, it is common to have a native and a target recipe (python and python-native) that use a common include file to specify the version, most of the build process, etc. That’s what you see in yocto-poky/meta/classes/python-dir.bbclass.

So, why isn’t python-native being used to run the python function in your recipe? A key insight is that you are using ‘addtask’ to tell BitBake to run this python function as a task. That means it is being run in the context of BitBake itself, not as a command executed by a recipe’s task. The sysroots described above don’t apply since a new python instance is not being executed. Instead, the recipe is run inside the existing BitBake python process. This is also how in-recipe python methods can alter BitBake’s variables and tasks (see BitBake User Manual Section 3.5.2 specifically that the ‘bb’ package is already imported and ‘d’ is available as a global variable). Since bitbake is executed directly from the command line and is a script, its shebang line tells us it will be run under python3.

If you want to run a python script as part of the recipe’s build process (perhaps a tool written in python), you’d execute that script from within a task:

do_check_version() {
  python -c '
import sys
print(sys.version)
'
}
addtask check_version after do_configure before do_compile

Since python is being executed within a task, the version of python in the selected sysroot will be used. The output will not be displayed on the console in BitBake’s output. Instead, it will be written to the task’s log file within the current build directory. If you need to do this, I highly suggest writing the script as a separate file that is included in the package rather than trying to contain it all inside the recipe.

—Brother Yocto