Bit-baking with soda

In one of my posts last year I mentioned that one can make automated comments to GitLab very easily with the right tooling - especially if they are coming from linting tools. So any author, reviewer and maintainer gets easy feedback as fast as possible on any proposed changes. This is super easy and very convenient when you're always doing a full build and every possible source in your project is actually checked. Why do we need something new, if that's working so well...? In the bitbake world that is different, we have powerful tools like sstate cache along other mechanisms to avoid exactly building everything from scratch all the time. This makes it tricky to map findings from the meta-sca layer (which fully supports sstate caching) to a pull or merge request, as we never can be sure to have the full picture. Moving from the outside, right into it So it was very clear that the commenting part of a CI pipeline needed to be done with the help of bitbake too... et voila scabot ...

 Let that title settle... and now we're getting back to a more serious issue :-). The issue When you're using bitbake layers you usually clone them on the fly when working in a cloud based setup, meaning a full clone of a repo, that could be highly expensive (just look at the size of the linux git repo for instance). As cloud based setups mostly don't supply a good way to sync those resources, unless you invent something yourself or pay for it, every bit counts. Not only as a matter of time but also as a matter of resulting cost. The layer meta-sca I maintain, has grown over the time quite much, so it became very very huge. Also because I made the mistake in the past to put large blobs (in this case tarballs) into the repository. I learned that lesson but I cannot undo it, as we all know each published git revision should stay untouched for all eternity. Mainly this is because of the linked-list nature of git - if I change one commit at the bottom I will alter any commit t...

Anyone that dealt with container engines came across go - a wonderful language, that was built to provide a right way of what C++ intended to do. The language itself is pretty straight forward and upstream poky support is given since ages... In the go world one would just run 1 2 go get github.com/foo/bar go build github.com/foo/bar and magically the go ecosystem would pull all the needed sources and build them into an executable. This is where the issues start... In the Openembedded world, one would have  one provider (aka recipe) for each dependency each recipe comes with a (remote) artifact (e.g. tarball, git repo, a.s.o.) which can be archived (so one can build the same software at a later point in time without any online connectivity) dedicated license information all this information is pretty useful when working is an environment (aka company) that has restrictions, such as reproducible builds license compliance security compliance (for instance no unpatched CVE) but whe...

Working in an environment with license restrictions is always a bit challenging. Recently while migrating a larger code base I encountered the following error message  1 Package gawk cannot be installed into the image because it has incompatible license(s): GPL-3.0 hummmm, where is that actually coming from? - a quick grep through my image recipes didn't reveal anything, so it has to be something pulled in by one of the packages or even further down the line. All the package data is available in a human readable format within your workspace, still it's hard to track the relations for the above mentioned issues... So I decided to write a small script which turns this into small and way better understandable trees in the console... Et voila, I present to you dot2tree , a small script which can turn different bitbake related sources into tree printouts in your console. Let's give it a try with the above shown error. As we all know, bitbake/poky creates an image manifest, ...

If you are working with are larger stack of recipes and update them frequently, you'll inevitably reach the point where a recipe becomes obsolete. Nothing in your stack will use it, so it basically is just a burden (and a potential hypothetical security risk). I had this situation with the insane amount of npm packages I maintain for my meta-sca layer. Those change nearly on a daily basis, with new dependencies coming in and replacing old dependencies. One could read all the change logs, but lets be honest, nobody does that but just for a small chosen few recipes - so the question remains... How to I identify obsolete recipes? Simple, by looking up all the dependencies of each recipe to another in a layer - kind of obvious isn't it :-). Lucky me, I don't have to do that manually, we are programmers, we automate stuff - so I did: the result can be found in my meta-buildutils layer - a small script called unused this one can be used without setting up bitbake at all, it ju...

This post is special, in the sense that it marks an important event in my life as a software developer. But let's start from the beginning... oelint-adv When you use bitbake/YOCTO on a frequent basis, you might have are checked out my linting tool for these file called  oelint-adv . This tool parses bitbake file and checks for several rules and best practices when it comes to working within this ecosystem. Initially I thought about using the original bitbake parser , which is available as open source licensed under GPL-2.0 - Unfortunately it requires more or less a whole workspace setup, including all layers a.s.o. As this was meant as a quick check tool to be used in CI or git pre-commit hooks, I was pretty sure that this wouldn't work, so I wrote my own parser for bitbake files, till then exclusively used with the linting tool. Scancode-toolkit All of a sudden, I got a few weeks back a ticket on my GitHub repo that somebody else had more or less the same issue - they wanted t...

When building a YOCTO based build in CI, normally you would get something like the following in your build log 1 2 3 4 5 6 7 Sstate summary: Wanted 0 Found 0 Missed 0 Current 60 (0% match, 100% complete) NOTE: Executing Tasks NOTE: Setscene tasks completed NOTE: Running task 272 of 272 (/build/my-recipes/python3-systemdlint-native/python3-systemdlint-native_1.1.12.bb:do_addto_recipe_sysroot) NOTE: recipe python3-systemdlint-native-1.1.12-r0: task do_addto_recipe_sysroot: Started NOTE: recipe python3-systemdlint-native-1.1.12-r0: task do_addto_recipe_sysroot: Succeeded NOTE: Tasks Summary: Attempted 272 tasks of which 271 didn't need to be rerun and all succeeded. depending on your build this could go on for hundreds or thousands of lines. If there is an error or a warning, typically one would use some sort of regular expression to filter out these items and present them somehow to the user. Unfortunately sometimes warnings and errors are multi-line and coding regex for ...

The YOCTO-project can do amazing things, but requires a very decent build machine, as by nature when you build everything from scratch it does require a lot of compilation. So the ultimate goal has to be to perform only the necessary steps in each run. Understanding task hashing The thing is that bitbake uses a task hashing to determine, which tasks (such as compilation, packaging, a.s.o.) are actually required to be performed. As tasks depend on each other, this information is also embedded into a hash, so the last task for a recipe is ultimately depending on the variable that are used for this specific task and every task before. You could visualize this by using a utility called bitbake-dumpsig , which produces output like this basewhitelist: {'SOURCE_DATE_EPOCH', 'FILESEXTRAPATHS', 'PRSERV_HOST', 'THISDIR', 'TMPDIR', 'WORKDIR', 'EXTERNAL_TOOLCHAIN', 'FILE', 'BB_TASKHASH', 'USER', 'BBSERVER...

In this post I'm telling my journey which started with my blog post about using Github Action as my main CI provider - in case you missed it see here  - on how to do a full yocto/poky build with just ~14GB of disk space Typically disk space is cheap nowadays and can be used without thinking too much about it - which leads to disk usage of 50GB and more for a yocto/poky build. So what are the options, when disk space becomes precious? Constraints, constraints, constraints While Github Actions is free for open source project (like mine) it is highly limited in regards of resources. Currently (2020/04/11) you get ( source )  2-core CPU 7 GB of RAM memory 14 GB of SSD disk space maximum of 6 hours per pipeline that's not very much when you think about doing a poky/yocto build on that, so every byte actually counts. Constant pain As the involved layer are constantly growing one has to care about every byte that could be saved, without giving up the overall a...

The 'a' in compliance stands for abstract things, that most developer try to avoid artificial created rules (as it's the nature of laws to be sometimes illogical;-)) an absolute must-have if you're working with open-source software. for me it additionally stand for automation potential Introducing the issue Just think of the following situation: a random guy, you never worked with before (and you don't know personally) tries to contribute a bitbake recipe for a new component. As we all know each recipe has a LICENSE entry, which offers a SPDX compatible representation of a LICENSE (or multiple licenses) applied to the source code, that the recipe offers build information for. So how can you be sure, that this setting is correct? Well you can scan the source code all by yourself and try to figure out under what license the source code is provided. For a single source repository, I guess that's  fairly easy, but time consuming. B...

What's the story? I guess most of you are familiar with the concept of merge or pull requests, so I won't go into details on these - but in case you missed it,  here is a brief description what GitLab thinks it is. There are basically two side involved into a merge request (or MR as the cool kids would say;-)) the one who actually provides the code change, lets call them devs the one who's maintaining the repository, where the change should be applied Both sides might have different objectives. The dev wants the code, she/he already wrote (hopefully tested) and streamlined, to be part of the repo upstream - as otherwise she/he could skipped all that work and enjoyed life (something we should all do from time to time). The maintainer usually lacks some implementation detail and is more keen on having the contributed code to be written according to the project style, the code being regression tested, covered by unit tests, so she/he can be sure that the fresh con...

When you're working with bitbake, especially when you're maintaining recipes, you might have asked yourself how do I know which recipes I need to update As we all read the YOCTO manual from beginning till the very end (as all good participants;-), you might be aware that you can check the `upstream` status of a package/recipe be running devtool check-upgrade-status myrecipe which returns (if there is an update available) INFO:myrecipe            0.12.8          0.12.9          None 297cb2458a96ea96d5e9d6ef38f1b7305c071f32  that means, currently you're running on version 0.12.8 and there is an update to 0.12.9 available at the defined source you're pulling your sources from. So far so easy - but wait Do I have to do that all manually, each and everytime? Automating things No, of course not. I'm using GitHub for hosting my sources, so I thought it would be quite convenient to let some ...