Bit-baking with soda

 Let that title settle... and now we're getting back to a more serious issue :-). The issue When you're using bitbake layers you usually clone them on the fly when working in a cloud based setup, meaning a full clone of a repo, that could be highly expensive (just look at the size of the linux git repo for instance). As cloud based setups mostly don't supply a good way to sync those resources, unless you invent something yourself or pay for it, every bit counts. Not only as a matter of time but also as a matter of resulting cost. The layer meta-sca I maintain, has grown over the time quite much, so it became very very huge. Also because I made the mistake in the past to put large blobs (in this case tarballs) into the repository. I learned that lesson but I cannot undo it, as we all know each published git revision should stay untouched for all eternity. Mainly this is because of the linked-list nature of git - if I change one commit at the bottom I will alter any commit t

Anyone that dealt with container engines came across go - a wonderful language, that was built to provide a right way of what C++ intended to do. The language itself is pretty straight forward and upstream poky support is given since ages... In the go world one would just run 1 2 go get github.com/foo/bar go build github.com/foo/bar and magically the go ecosystem would pull all the needed sources and build them into an executable. This is where the issues start... In the Openembedded world, one would have  one provider (aka recipe) for each dependency each recipe comes with a (remote) artifact (e.g. tarball, git repo, a.s.o.) which can be archived (so one can build the same software at a later point in time without any online connectivity) dedicated license information all this information is pretty useful when working is an environment (aka company) that has restrictions, such as reproducible builds license compliance security compliance (for instance no unpatched CVE) but when us

Working in an environment with license restrictions is always a bit challenging. Recently while migrating a larger code base I encountered the following error message  1 Package gawk cannot be installed into the image because it has incompatible license(s): GPL-3.0 hummmm, where is that actually coming from? - a quick grep through my image recipes didn't reveal anything, so it has to be something pulled in by one of the packages or even further down the line. All the package data is available in a human readable format within your workspace, still it's hard to track the relations for the above mentioned issues... So I decided to write a small script which turns this into small and way better understandable trees in the console... Et voila, I present to you dot2tree , a small script which can turn different bitbake related sources into tree printouts in your console. Let's give it a try with the above shown error. As we all know, bitbake/poky creates an image manifest,

If you are working with are larger stack of recipes and update them frequently, you'll inevitably reach the point where a recipe becomes obsolete. Nothing in your stack will use it, so it basically is just a burden (and a potential hypothetical security risk). I had this situation with the insane amount of npm packages I maintain for my meta-sca layer. Those change nearly on a daily basis, with new dependencies coming in and replacing old dependencies. One could read all the change logs, but lets be honest, nobody does that but just for a small chosen few recipes - so the question remains... How to I identify obsolete recipes? Simple, by looking up all the dependencies of each recipe to another in a layer - kind of obvious isn't it :-). Lucky me, I don't have to do that manually, we are programmers, we automate stuff - so I did: the result can be found in my meta-buildutils layer - a small script called unused this one can be used without setting up bitbake at all, it ju

This post is special, in the sense that it marks an important event in my life as a software developer. But let's start from the beginning... oelint-adv When you use bitbake/YOCTO on a frequent basis, you might have are checked out my linting tool for these file called  oelint-adv . This tool parses bitbake file and checks for several rules and best practices when it comes to working within this ecosystem. Initially I thought about using the original bitbake parser , which is available as open source licensed under GPL-2.0 - Unfortunately it requires more or less a whole workspace setup, including all layers a.s.o. As this was meant as a quick check tool to be used in CI or git pre-commit hooks, I was pretty sure that this wouldn't work, so I wrote my own parser for bitbake files, till then exclusively used with the linting tool. Scancode-toolkit All of a sudden, I got a few weeks back a ticket on my GitHub repo that somebody else had more or less the same issue - they wanted t

When building a YOCTO based build in CI, normally you would get something like the following in your build log 1 2 3 4 5 6 7 Sstate summary: Wanted 0 Found 0 Missed 0 Current 60 (0% match, 100% complete) NOTE: Executing Tasks NOTE: Setscene tasks completed NOTE: Running task 272 of 272 (/build/my-recipes/python3-systemdlint-native/python3-systemdlint-native_1.1.12.bb:do_addto_recipe_sysroot) NOTE: recipe python3-systemdlint-native-1.1.12-r0: task do_addto_recipe_sysroot: Started NOTE: recipe python3-systemdlint-native-1.1.12-r0: task do_addto_recipe_sysroot: Succeeded NOTE: Tasks Summary: Attempted 272 tasks of which 271 didn't need to be rerun and all succeeded. depending on your build this could go on for hundreds or thousands of lines. If there is an error or a warning, typically one would use some sort of regular expression to filter out these items and present them somehow to the user. Unfortunately sometimes warnings and errors are multi-line and coding regex for