Keeping it fresh - the lazy way

When you're working with bitbake, especially when you're maintaining recipes, you might have asked yourself

how do I know which recipes I need to update

As we all read the YOCTO manual from beginning till the very end (as all good participants;-), you might be aware that you can check the `upstream` status of a package/recipe be running

devtool check-upgrade-status myrecipe

which returns (if there is an update available)

INFO:myrecipe 0.12.8 0.12.9 None 297cb2458a96ea96d5e9d6ef38f1b7305c071f32

that means, currently you're running on version 0.12.8 and there is an update to 0.12.9 available at the defined source you're pulling your sources from.

So far so easy - but wait

Do I have to do that all manually, each and everytime?

Automating things

No, of course not.
I'm using GitHub for hosting my sources, so I thought it would be quite convenient to let some pipeline create an issue in case a new update was found.

Finding all recipes to maintain

As I'm maintaining a complete layer with plenty of recipes in it, I needed to get a comprehensive list of all recipes (without the bbappend-files) to check.

So I ran

bitbake-layers show-recipes -l mylayer -b

which prints all the recipes which belong to `mylayer` (-l mylayer) without any fancy output formatting (-b).
This resulted in something like this

stank-native:
mylayer 0.0.13
stylelint-native:
mylayer 13.0.0
sudokiller:
mylayer 20200206
tbb-native:
mylayer 20170412+AUTOINC+75224c4c1f
textlint-native:
mylayer 11.6.3
tiger:
mylayer 20190819
tscancode-native:
mylayer 20190324
upc:
mylayer 20170904

not quite what I wanted, but a good start. With some piping magic on the shell

bitbake-layers show-recipes -l mylayer -b | grep ":" | tr -d ':' | tr '\n' ' '

I finally got a list of recipes, which could be directly put into the argument list of the devtool call

devtool check-upgrade-status $(bitbake-layers show-recipes -l mylayer -b | grep ":" | tr -d ':' | tr '\n' ' ')

Grepping the return

I felt I was on a good way by that time, so it was time to grep the needed information, so I piped the output into a python script, which should transform the found update information into GitHub issues using the github3 API and some regex. I used

^INFO:\s+(?P<recipe>[A-Za-z0-9\+\.\-_]+)\s+(?P<curversion>[A-Za-z0-9\.\-_]+)\s+(?P<nextversion>[A-Za-z0-9\.\-_]+)\s+.*

as my regex.

Things are weird

I ran the scripts and I got some update information but also things like

INFO:myrecipe 0.12.8 UNKNOWN_BROKEN None 297cb2458a96ea96d5e9d6ef38f1b7305c071f32

a lot.
What's the background of that?

`UNKNOWN_BROKEN` is the bitbake way of saying I can't check the upstream source.

There are plenty of reasons for this behavior, for example

recipe references a direct source revision and not a tag
naming convention of the version doesn't match the 'standard' way
PV isn't set in SRC_URI

So it was time to edit some recipes to fix that

Fixing recipes where PV wasn't set

For instance there was the following set in a recipe

SRC_URI = "git://github.com/fancy/project.git;protocol=https;tag=v1.2.3"

after changing it to

SRC_URI = "git://github.com/fancy/project.git;protocol=https;tag=v${PV}"

it worked out of the box.
Background is that bitbake needs something to alter/compare to - and as it doesn't recognize by itself, which part of the tag-entry is a version and which are just prefix/suffix, we had to 'mark' that by using the ${PV} variable.

Fixing recipes where direct revision is used

Just think of a typical recipe like this

SRC_URI = "git://git.savannah.gnu.org/git/tiger.git;protocol=https"
SRCREV = "1ae0a295bcd6fe4e772d74b206eb10a6fb4d9a0e"

By default bitbake tries to check on versions and not commit hashes - this has to be enabled explicitly by adding

UPSTREAM_CHECK_COMMITS = "1"

to the recipe.

Fixing recipes with non-standard naming scheme

When checking resources fetched by http/https/ftp bitbake does a directory listing on the server and applies an regex to the return.

You can instruct bitbake to perform checks by setting the variable

UPSTREAM_CHECK_REGEX = "(?P<pver>(\d+[\.\-_]*)+)"

which then would extract all strings matching the regex '(\d+[\.\-_]*)+' and compare all of them to get the `highest`/`latest` version.

This mostly works, but I came across a ruby recipe, where the check always failed.

I was pulling sources from rubygems.com, e.g. for 'example' the source is 'https://rubygems.org/gems/example-1.0.0.gem'.

You can also get an overview of all versions of that recipe by calling 'https://rubygems.org/gems/example/versions' - output is some HTML, so no easy parseable list.

So I took a look into the HTML

<div class="versions">
<ul class="t-list__items">
<li class="gem__version-wrap">
<a class="t-list__item" href="/gems/example/versions/1.0.2">1.0.2</a>
- March 19, 2011
(17,5 KB)
</li>
<li class="gem__version-wrap">
<a class="t-list__item" href="/gems/example/versions/1.0">1.0</a>
- October 12, 2010
(3 KB)
yanked
</li>
<li class="gem__version-wrap">
<a class="t-list__item" href="/gems/example/versions/1.0.0">1.0.0</a>
- March 19, 2011
(6,5 KB)
</li>
</ul>
</div>

there I noticed the strings like '/gems/example/versions/1.0.2' and '/gems/example/versions/1.0.0'.
From here on it's easy, right?

UPSTREAM_CHECK_URI ?= "https://rubygems.org/gems/${BPN}/versions"
UPSTREAM_CHECK_REGEX ?= "/gems/${BPN}/versions/(?P<pver>(\d+[\.\-_]*)+)"

just alter the URL where to check for updates and setup an appropriate regex and you're done.

Further fine tuning

Now I came across some recipes, where there were multiple major branches of the tool available and I just wanted to get the updates for my used branch.
This is where, again, I needed to edit the recipe, but as I was using git already the 'UPSTREAM_CHECK_REGEX' had no effect - I had to use 'UPSTREAM_CHECK_GITTAGREGEX'

E.g. for my tool there are the major branches 2.x and 3.x existing in the original source repo - for compatibility I need to stay on 2.x releases, but the upgrade check always returns 3.x versions.

Consequently I needed to configure

UPSTREAM_CHECK_GITTAGREGEX = "(?P<pver>2\.\d+\.*\d*)"

which did the trick.

Strip off the garbage. Period.

Another thing I noticed is that false positive issues where created, where the version was the same as before but with a trailing ".".
When running the check manually following output was shown

INFO:myrecipe 0.12.8 0.12.8. None 297cb2458a96ea96d5e9d6ef38f1b7305c071f32

So I needed to add a sanity check to my script which strips trailing period characters, compare them again before filing an issue.
I guess that somehow a bug of bitbake and I'll surely spend some minutes in the near future to further check on that...

There is always a downside

The job was running and I was happy to spend a little more time on different things.
After some time I was wondering that the script returned so less new issues while I was getting quite a lot GitHub notification about new releases by mail.

So it was again time to investigate. After I dumped all output of the tool it seemed like the actual check had never been performed for the suspected recipe - at least I didn't see anything.

When running the same check in a single instance (without all the other recipes involved) everything was fine - Very strange I just thought.

So had to go back one step.
Instead of the mentioned script

devtool check-upgrade-status $(bitbake-layers show-recipes -l mylayer -b | grep ":" | tr -d ':' | tr '\n' ' ')

it had to be

for recipe in $(bitbake-layers show-recipes -l mylayer -b | grep ":" | tr -d ':' | tr '\n' ' '); do devtool check-upgrade-status ${recipe}done

which is working for me.

Conclusion

Bitbake offers interesting tools for automating things, which makes maintenance quite charming, once you figured out how things are working. So no need to rely on email notification for various projects anymore.

I further improved the script and the GitHub integration to make it even more convenient - if you like to see my current setup have a look here.
Feel free to use that as a basis for your own implementation and don't forget to comment on this post.

Enjoy.

Update

As Alexander Kanavin pointed out, a even more sophisticated approach is existing at http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/, which uses the full power of devtool - so in the end you will get patches send by mail to the setup maintainers.
Really nice solution, be sure to check it out as well, if it fits your project's needs.

Making go not a no-go

Anyone that dealt with container engines came across go - a wonderful language, that was built to provide a right way of what C++ intended to do. The language itself is pretty straight forward and upstream poky support is given since ages... In the go world one would just run 1 2 go get github.com/foo/bar go build github.com/foo/bar and magically the go ecosystem would pull all the needed sources and build them into an executable. This is where the issues start... In the Openembedded world, one would have one provider (aka recipe) for each dependency each recipe comes with a (remote) artifact (e.g. tarball, git repo, a.s.o.) which can be archived (so one can build the same software at a later point in time without any online connectivity) dedicated license information all this information is pretty useful when working is an environment (aka company) that has restrictions, such as reproducible builds license compliance security compliance (for instance no unpatched CVE) but whe...

Bit-baking with soda

Dieses Blog durchsuchen