Sharing is caring... about task hashes

The YOCTO-project can do amazing things, but requires a very decent build machine, as by nature when you build everything from scratch it does require a lot of compilation. So the ultimate goal has to be to perform only the necessary steps in each run.

Understanding task hashing

The thing is that bitbake uses a task hashing to determine, which tasks (such as compilation, packaging, a.s.o.) are actually required to be performed.
As tasks depend on each other, this information is also embedded into a hash, so the last task for a recipe is ultimately depending on the variable that are used for this specific task and every task before.

You could visualize this by using a utility called bitbake-dumpsig, which produces output like this

basewhitelist: {'SOURCE_DATE_EPOCH', 'FILESEXTRAPATHS', 'PRSERV_HOST', 'THISDIR', 'TMPDIR', 'WORKDIR', 'EXTERNAL_TOOLCHAIN', 'FILE', 'BB_TASKHASH', 'USER', 'BBSERVER', 'CCACHE_NOHASHDIR', 'CCACHE_DIR', 'ERROR_QA', 'STAMPS_DIR', 'SHELL', 'SSTATE_HASHEQUIV_METHOD', 'WARN_QA', 'extend_recipe_sysroot', 'LOGNAME', 'HOME', 'CCACHE_TOP_DIR', 'BBPATH', 'BB_HASHSERVE', 'SSTATE_PKGARCH', 'BB_UNIHASH', 'STAMPCLEAN', 'DEPLOY_DIR', 'PWD', 'BB_LIMITEDDEPS', 'CCACHE', 'FILE_DIRNAME', 'SSTATE_DIR', 'PRSERV_DUMPFILE', 'COREBASE', 'STAGING_DIR_TARGET', 'BB_WORKERCONTEXT', 'BUILD_ARCH', 'STAGING_DIR_HOST', 'TERM', 'SDKPKGSUFFIX', 'SSTATE_HASHEQUIV_OWNER', 'LICENSE_PATH', 'PATH', 'PRSERV_LOCKDOWN', 'FILESPATH', 'PKGDATA_DIR', 'PARALLEL_MAKE', 'SSTATE_HASHEQUIV_REPORT_TASKDATA', 'PRSERV_DUMPDIR', 'DL_DIR'}
taskwhitelist: None
Task dependencies: ['ACLOCALDIR', 'ACLOCALEXTRAPATH', 'AR', 'AS', 'AUTOTOOLS_AUXDIR', 'AUTOTOOLS_SCRIPT_PATH', 'B', 'BPN', 'BUILD_AR', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_FC', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_OS', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'BUILD_SYS', 'BUILD_VENDOR', 'CACHED_CONFIGUREVARS', 'CC', 'CCLD', 'CC_FOR_BUILD', 'CFLAGS', 'CFLAGS_FOR_BUILD', 'CLEANBROKEN', 'CONFIGUREOPTS', 'CONFIGUREOPT_DEPTRACK', 'CONFIGURESTAMPFILE', 'CONFIGURE_SCRIPT', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CPPFLAGS_FOR_BUILD', 'CPP_FOR_BUILD', 'CXX', 'CXXFLAGS', 'CXXFLAGS_FOR_BUILD', 'CXX_FOR_BUILD', 'DEBUG_BUILD', 'DEPENDS', 'DISABLE_STATIC', 'EXTRA_AUTORECONF', 'EXTRA_NATIVE_PKGCONFIG_PATH', 'EXTRA_OECONF', 'EXTRA_OEMAKE', 'FC', 'HOST_ARCH', 'HOST_OS', 'HOST_PREFIX', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LC_ALL', 'LD', 'LDFLAGS', 'LDFLAGS_FOR_BUILD', 'LD_FOR_BUILD', 'LOGFIFO', 'MAKE', 'MLPREFIX', 'NATIVE_PACKAGE_PATH_SUFFIX', 'NM', 'OBJCOPY', 'OBJDUMP', 'P', 'PACKAGECONFIG', 'PACKAGECONFIG_CONFARGS', 'PERL_HASH_SEED', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PN', 'PSEUDO_DISABLED', 'PV', 'PYTHONHASHSEED', 'QA_LOGFILE', 'QA_SANE', 'RANLIB', 'READELF', 'RECIPE_SYSROOT_NATIVE', 'S', 'SITECONFIG_SYSROOTCACHE', 'SITEINFO_EXTRA_DATAFUNCS', 'SPECIAL_PKGSUFFIX', 'STAGING_BASE_LIBDIR_NATIVE', 'STAGING_BINDIR_NATIVE', 'STAGING_DATADIR', 'STAGING_DATADIR_NATIVE', 'STAGING_DIR_NATIVE', 'STAGING_ETCDIR_NATIVE', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STAGING_SBINDIR_NATIVE', 'STRINGS', 'STRIP', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TZ', 'UNKNOWN_CONFIGURE_WHITELIST', 'acpaths', 'append_libtool_sysroot', 'autotools_aclocals', 'autotools_do_configure', 'autotools_postconfigure', 'autotools_preconfigure', 'base_bindir', 'base_libdir', 'base_libdir_native', 'base_prefix', 'base_sbindir', 'bbfatal', 'bbfatal_log', 'bbnote', 'bbwarn', 'bindir', 'bindir_native', 'datadir', 'datadir_native', 'die', 'do_qa_configure', 'docdir', 'exec_prefix', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'nonarch_base_libdir', 'nonarch_libdir', 'oe_runconf', 'oe_runmake', 'oe_runmake_call', 'oldincludedir', 'package_qa_handle_error', 'package_qa_write_error', 'prefix', 'prefix_native', 'root_prefix', 'sbindir', 'sbindir_native', 'servicedir', 'sharedstatedir', 'siteinfo_data', 'siteinfo_data_for_machine', 'siteinfo_get_files', 'sysconfdir', 'sysconfdir_native', 'systemd_system_unitdir', 'systemd_unitdir', 'systemd_user_unitdir']
basehash: 0b85a6b7be511f5446ffbe861bf80b7c6b0eb2664576170cb40ffa3d550eedf7
List of dependencies for variable STAGING_BINDIR_NATIVE is {'STAGING_DIR_NATIVE', 'bindir_native'}
List of dependencies for variable STAGING_DATADIR is {'datadir'}
List of dependencies for variable STAGING_DATADIR_NATIVE is {'STAGING_DIR_NATIVE', 'datadir_native'}
Variable siteinfo_get_files value is def siteinfo_get_files(d, sysrootcache = False):
    sitedata = siteinfo_data(d)
    sitefiles = ""
    for path in d.getVar("BBPATH").split(":"):
        for element in sitedata:
            filename = os.path.join(path, "site", element)
            if os.path.exists(filename):
                sitefiles += filename + " "
    if not sysrootcache:
        return sitefiles
    # Now check for siteconfig cache files in sysroots
    path_siteconfig = d.getVar('SITECONFIG_SYSROOTCACHE')
    if path_siteconfig and os.path.isdir(path_siteconfig):
        for i in os.listdir(path_siteconfig):
            if not i.endswith("_config"):
                continue
            filename = os.path.join(path_siteconfig, i)
            sitefiles += filename + " "
    return sitefiles
Computed task hash is e19d2b0a0554f626b8a966347d66e6f4a501c4a7e2d24064c101d8700aaea4a3

as you see each task and each variable and their reference to each other is known to bitbake.
Ultimately bitbake does compare the taskhash of the currently know build fragments with the hash that is calculated from the current recipe information - if they don't match the task needs to be run.

Sharing is a good thing

As you may know you can share build fragments between build hosts, that's called sstate-mirror.

Also you may know that each build host is somehow unique, as they might be different OS versions or even distributions - so what happens when any of the variables are pointing to something that is unique to the build host...?

You might have guessed it: the sstate cache can then NOT be reused, leading to longer build times.

Fighting the (change of a) hash

So these variables have to be marked as host specific.

You can use settings like BB_HASHCONFIG_WHITELIST or BB_HASHBASE_WHITELIST to globally disable the usage for calculating a task hash - that would be the brute force way.

It would be better to address this issues on a task level - for this bitbake offers the task-flag vardepsexclude to (you might have guessed it) exclude a variable from being used for calculating the task hash.

Identifying these variables that might cause such a corruption is not that easy and requires a lot of "inside" knowledge of the poky/bitbake internals, so I coded a small piece of code to identify variables used in tasks that are either known to be host specific or are pointing to hard coded absolute paths - both will cause hash differences between different build hosts.

Example time

lets use a fairly easy example

MYVAR = "${TOPDIR}/foo"
ABC = "1"
do_magictask() {
    echo ${ABC} > ${MYVAR}
}

this task creates a file in some folder - so far so good - but TOPDIR is an absolute path, so on another host this task may be re-executed as  TOPDIR is some other path, although the outcome of the task is the same - and that's all that matters.

Using my hashdog.bbclass would then produce a warning on the build console like this

WARNING: myrecipe-1.0-r0 do_hashdog: Task 'do_magictask' uses TOPDIR with the value of '/my/build/path' - this could lead to sstate corruptions.
Consider adding 'do_magictask[vardepsexclude] += "TOPDIR"' to the recipe

so to fix the possible corruption one just have to add

MYVAR = "${TOPDIR}/foo"
ABC = "1"
do_magictask() {
    echo ${ABC} > ${MYVAR}
}
do_magictask[vardepsexclude] += "TOPDIR"

and on every build host the same hashes will be produced, leading to less rebuilding and overall better performance of your bitbake projects across multiple build hosts

Conclusion

Understanding bitbake hashing will help a lot turning overall build times down in your project - and identifying code that corrupts those hashes should be a high priority item.

If you liked this helper class be sure to check the other build utilities I've written for bitbake at meta-buildutils