Dear Brother Yocto,

I have two images: production-image.bb and manufacturing-image.bb. The production-image includes a recipe, production-systemd-scripts, and the manufacturing-image includes a different recipe, manufacturing-systemd-scripts. The two recipes have different versions of a systemd service, startup-ui.service. One starts the production app and the other does not. If I start with a clean tmp directory they both build fine, but if I build one first and then build the other, I get an error:

$ bitbake production-image
...
Summary: There was 1 WARNING message shown.
(it works)
$ bitbake manufacturing-image
...
ERROR: manufacturing-systemd-scripts-0.0.1+gitAUTOINC+60423a2c5f-r0 do_populate_sysroot: The recipe manufacturing-systemd-scripts is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
/home/user/fsl-release-bsp/x11_ull_build/tmp/sysroots/imx6ull14x14evk/lib/systemd/system/startup-ui.service
Matched in b'manifest-imx6ull14x14evk-pika.populate_sysroot'
Please verify which recipe should provide the above files.
The build has stopped as continuing in this scenario WILL break things, if not now, possibly in the future (we've seen builds fail several months later). If the system knew how to recover from this automatically it would however there are several different scenarios which can result in this and we don't know which one this is. It may be you have switched providers of something like virtual/kernel (e.g. from linux-yocto to linux-yocto-dev), in that case you need to execute the clean task for both recipes and it will resolve this error. It may be you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning those recipes should again resolve this error however switching DISTRO_FEATURES on an existing build directory is not supported, you should really clean out tmp and rebuild (reusing sstate should be safe). It could be the overlapping files detected are harmless in which case adding them to SSTATE_DUPWHITELIST may be the correct solution. It could also be your build is including two different conflicting versions of things (e.g. bluez 4 and bluez 5 and the correct solution for that would be to resolve the conflict. If in doubt, please ask on the mailing list, sharing the error and filelist above.
ERROR: manufacturing-systemd-scripts-0.0.1+gitAUTOINC+60423a2c5f-r0 do_populate_sysroot: If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.
ERROR: manufacturing-systemd-scripts-0.0.1+gitAUTOINC+60423a2c5f-r0 do_populate_sysroot: Function failed: sstate_task_postfunc

The recipe production-systemd-scripts is not in the image I asked to build. Why doesn’t bitbake remove the output of unused recipes from tmp/sysroots before I get this error?

—Conflicted about sysroots

Dear Conflicted,

When I first started with Yocto, I assumed that a recipe is built in isolation and should have no side-effects on other packages. Of course, that isn’t strictly true as a recipe has access to all of the files installed by its dependencies (both build-time and run-time). If I were designing a new build system, I’d probably implement the build steps as:

1. Create an empty directory
2. Install all dependent packages into that directory
3. Enter this directory as a “chroot”
4. Build the recipe
5. Package the results

The Problem

On the surface, this seems great: each package only has access to the files provided by its dependency chain (note that transitive dependencies are still a problem). There is a serious problem lurking under the surface though. Assume I have two recipes: A and B. A and B both depend on a common library C but they require different, incompatible major versions. Let’s say A depends on C v1.0.0 while B requires C v2.0.0. The builds of A and B will succeed because each build environment includes only its own dependency chain. When the resulting packages are installed, however, one of the packages will be broken. Why? Because the two major versions of C use the same library and header names. When C v2.0.0 is installed after C v1.0.0, B will work just fine but A will either fail to find some symbols or, worse, start up and then crash due to calling functions with the wrong arguments.

“Wait!,” I hear you screaming, “this is already handled by library major versions using different library names.” That’s true, most major libraries have encountered this exact problem and worked around it by effectively creating a whole new project for the new major version. That’s only a solution if you know of the hazard though. What about smaller libraries with developers that haven’t encountered this situation before? They can wreak havoc on an OS build. This type of conflict happens often enough that every OS build system I’ve encountered has some type of warning or error when this either happens or is possible.

Enter the sysroot

How does Yocto deal with this situation? As the error messages you encountered indicate, it has something to do with sysroots. Before I dive into details, please note that Yocto’s sysroot behavior changed in 2.5 (Sumo). I’ll be referencing documentation from 2.5 because it has much better descriptions of the build process. When 2.5 behavior differs from prior releases, I’ll indicate this and explain the differences.

Section 4.3 of Yocto Project Overview and Concepts Manual provides a thorough introduction to Yocto’s various build stages, their inputs, and their outputs. While it does discuss sysroots, it doesn’t provide a succinct description of what they are used for. For that, we need to look at Yocto Reference Manual’s staging.bbclass section. The rough concept is that sysroots are the mechanism used to share files between recipes.

As described in Section 4.3.5.3 of Yocto Project Overview and Concepts Manual, a recipe’s do_install task is responsible for installing a recipe’s build products into the recipe’s staging directory ($D). Why doesn’t it install directly into the sysroot? Two reasons.

First, the sysroot already contains the files installed by every dependency. While this might work to build a single image, isolating the recipe’s files allows the recipe’s output to be cached, bundled into a package (deb, rpm, opkg, etc), and used in multiple images that may have different collections of recipes installed.

Second, instead of simply creating a single package containing every file installed by the recipe, Yocto splits the installed files into multiple packages such as -dev and -dbg to allow finer-grain control over image contents. That way a production image only includes executables while the matching debug image can include symbol files and the developer image has all the headers and static libraries. In addition to the packages used to construct images, the do_populate_sysroot task copies files and directories specified by the SYSROOT_DIR* variables to a staging area that is made available at build time of other recipe’s that depend on this recipe. Note that this means that a recipe that depends on another recipe, A, may have a different set of files available than if you installed an SDK that includes the A-dev package. Put another way, sysroots are how files are shared between recipes during a build of an image while -dev packages are used to construct the final contents of an SDK image.

But that doesn’t seem to cause a problem…

As mentioned in The Problem, sysroot problems are rarely encountered during the build or install phases of a recipe. There are two places after building where sysroots can run into problems: during staging (do_populate_sysroot) and during setup as a dependency of another recipe (do_prepare_recipe_sysroot).

During staging, the files matching SYSROOT_DIRS* are copied into a sysroot. If a dependency has already staged a file with the same path, it would be overwritten. Because this is likely to be a Very Bad Thing™, Yocto raises a warning. I would expect this to be uncommon except for one big caveat: before Yocto 2.5, a single, global sysroot was used for all recipes. Instead of file path uniqueness within a dependency chain being required, globally unique file paths were necessary. As illustrated in the figure below, Yocto 2.5 generates a sysroot for each recipe based solely on its dependencies which avoids path collisions from recipes with independent dependency chains.

The second place where collisions can happen is when constructing a recipe’s initial sysroot based on its dependency chain. As described in The Problem, the individual dependencies might build without issue but still result in a conflict when they are installed together. Going back to that example, an image D that depends on both A and B will result in two different versions of C being installed that overwrite each other. Note that this only happens in Yocto 2.5 and later. In earlier versions of Yocto, these collisions would be encountered during staging into the global sysroot.

Wrapping up

So, what’s going on in your build? You described two recipes installing a file to the same path but those recipes are never used together in a single image. This should work in general and it will work in Yocto 2.5 and later. For earlier versions, the global sysroot will cause a conflict.

A common workaround is to use a single image with custom IMAGE_FEATURES to select which recipe is included. Any modifications to IMAGE_FEATURES or DISTRO_FEATURES is a signal that a new build directory should be used. The downside is that the production and manufacturing images will be built independently and may not be representative of each other.

Another approach is to use separate systemd unit names for production-systemd-scripts and manufacturing-systemd-scripts. Assuming your main concern is controlling which services are started during boot, the names of the units shouldn’t matter. In fact, a recipe named production-systemd-scripts points to a bigger, conceptual issue: systemd unit files should be coupled with the service that the unit controls. That way, the systemd units started reflect the services available rather than a desired behavior. Behaviors should be separate recipes that install a single systemd unit whose presence triggers that behavior. If you have a systemd unit that automatically starts manufacturing tests, put that in an appropriately-named recipe separate from the tests themselves. That way, the tests can be installed on top of a production image without triggering the manufacturing workflow. Even better is if you create your own pseudo-targets for these behaviors and then use WantedBy and Conflicts to allow switching between modes by activating or deactivating the psuedo-targets. That gets you a single image that can be used for both manufacturing test and production.

—Brother Yocto