Bit-baking with soda

In one of my posts last year I mentioned that one can make automated comments to GitLab very easily with the right tooling - especially if they are coming from linting tools. So any author, reviewer and maintainer gets easy feedback as fast as possible on any proposed changes. This is super easy and very convenient when you're always doing a full build and every possible source in your project is actually checked. Why do we need something new, if that's working so well...? In the bitbake world that is different, we have powerful tools like sstate cache along other mechanisms to avoid exactly building everything from scratch all the time. This makes it tricky to map findings from the meta-sca layer (which fully supports sstate caching) to a pull or merge request, as we never can be sure to have the full picture. Moving from the outside, right into it So it was very clear that the commenting part of a CI pipeline needed to be done with the help of bitbake too... et voila scabot ...

 Let that title settle... and now we're getting back to a more serious issue :-). The issue When you're using bitbake layers you usually clone them on the fly when working in a cloud based setup, meaning a full clone of a repo, that could be highly expensive (just look at the size of the linux git repo for instance). As cloud based setups mostly don't supply a good way to sync those resources, unless you invent something yourself or pay for it, every bit counts. Not only as a matter of time but also as a matter of resulting cost. The layer meta-sca I maintain, has grown over the time quite much, so it became very very huge. Also because I made the mistake in the past to put large blobs (in this case tarballs) into the repository. I learned that lesson but I cannot undo it, as we all know each published git revision should stay untouched for all eternity. Mainly this is because of the linked-list nature of git - if I change one commit at the bottom I will alter any commit t...

When building a YOCTO based build in CI, normally you would get something like the following in your build log 1 2 3 4 5 6 7 Sstate summary: Wanted 0 Found 0 Missed 0 Current 60 (0% match, 100% complete) NOTE: Executing Tasks NOTE: Setscene tasks completed NOTE: Running task 272 of 272 (/build/my-recipes/python3-systemdlint-native/python3-systemdlint-native_1.1.12.bb:do_addto_recipe_sysroot) NOTE: recipe python3-systemdlint-native-1.1.12-r0: task do_addto_recipe_sysroot: Started NOTE: recipe python3-systemdlint-native-1.1.12-r0: task do_addto_recipe_sysroot: Succeeded NOTE: Tasks Summary: Attempted 272 tasks of which 271 didn't need to be rerun and all succeeded. depending on your build this could go on for hundreds or thousands of lines. If there is an error or a warning, typically one would use some sort of regular expression to filter out these items and present them somehow to the user. Unfortunately sometimes warnings and errors are multi-line and coding regex for ...

The YOCTO-project can do amazing things, but requires a very decent build machine, as by nature when you build everything from scratch it does require a lot of compilation. So the ultimate goal has to be to perform only the necessary steps in each run. Understanding task hashing The thing is that bitbake uses a task hashing to determine, which tasks (such as compilation, packaging, a.s.o.) are actually required to be performed. As tasks depend on each other, this information is also embedded into a hash, so the last task for a recipe is ultimately depending on the variable that are used for this specific task and every task before. You could visualize this by using a utility called bitbake-dumpsig , which produces output like this basewhitelist: {'SOURCE_DATE_EPOCH', 'FILESEXTRAPATHS', 'PRSERV_HOST', 'THISDIR', 'TMPDIR', 'WORKDIR', 'EXTERNAL_TOOLCHAIN', 'FILE', 'BB_TASKHASH', 'USER', 'BBSERVER...

In this post I'm telling my journey which started with my blog post about using Github Action as my main CI provider - in case you missed it see here  - on how to do a full yocto/poky build with just ~14GB of disk space Typically disk space is cheap nowadays and can be used without thinking too much about it - which leads to disk usage of 50GB and more for a yocto/poky build. So what are the options, when disk space becomes precious? Constraints, constraints, constraints While Github Actions is free for open source project (like mine) it is highly limited in regards of resources. Currently (2020/04/11) you get ( source )  2-core CPU 7 GB of RAM memory 14 GB of SSD disk space maximum of 6 hours per pipeline that's not very much when you think about doing a poky/yocto build on that, so every byte actually counts. Constant pain As the involved layer are constantly growing one has to care about every byte that could be saved, without giving up the overall a...

This time I want to write something about an issue everybody maintaining a git repo might faced already - CI. In theory every push and every pull request should have been build with all supported layer-versions... well in theory. The issue If you have a local setup it's sometimes hard to switch layer-version - I agree the usage of repo  is highly recommended here, as it simplifies such work heavily. Nevertheless you might need multiple work spaces, which all need a lot of disk space. Roughly calculated you can expect ~50GB of data per architecture/distro-combination without the usage of " rm_work "-bbclass, round about 15GB if you're using it. So if you decided to support >3 layer versions of YOCTO, it's a lot of space blocked for a lot of redundant data. Not to mention that you need to build everything now and then to get results, if your code is still working or not. Solution: CI This is where you pick a CI provider - Jenkins immediately comes in...