Using a bitbake CI - For Free

This time I want to write something about an issue everybody maintaining a git repo might faced already - CI.

In theory every push and every pull request should have been build with all supported layer-versions... well in theory.

The issue

If you have a local setup it's sometimes hard to switch layer-version - I agree the usage of repo is highly recommended here, as it simplifies such work heavily.
Nevertheless you might need multiple work spaces, which all need a lot of disk space.

Roughly calculated you can expect ~50GB of data per architecture/distro-combination without the usage of "rm_work"-bbclass, round about 15GB if you're using it.

So if you decided to support >3 layer versions of YOCTO, it's a lot of space blocked for a lot of redundant data. Not to mention that you need to build everything now and then to get results, if your code is still working or not.

Solution: CI

This is where you pick a CI provider - Jenkins immediately comes into my mind - to do all the work for you.
When you are with a company you normally have some on-premise hardware where to run such a service - I was lacking of some capable hardware, as this was my spare time project.

So I googled around for a service which does CI for me, something not that pricy, which means free would be best.

Travis CI

As my code is in a public repository, my first pick was Travis CI, which is offering CI service for free for all public repositories - Great!

I added a file called ".travis.yml" to my repository root, pushed it and voila a build ran immediately after my push - I went away to have a coffee or two and as I came back the following notification was in my inbox

Build #1 Failed

A larger stack of limitations

Damn - I had a look at the log and it said

The job exceeded the maximum time limit for jobs, and has been terminated.

which reminded myself of reading the manual first ((sic) - who does this???) - and there it was: the free service is limited in the following ways

• maximum job runtime = 50min (including setup)
• 2 vCPUs

Yes - this won't work with bitbake/YOCTO, as a typical "from-scratch" build with just 2 CPU
will take at least 4h.
So I continued to read through the manual of Travis CI.

Digging deeper in Travis CI

Next I found that Travis CI offers a caching mechanism where you can save build artifacts for upcoming builds - and I though why not cache the bitbake downloads and sstate cache (like suggested here), auto terminate the build after ~40min without issuing an error (like suggest here) and let the build iterate more than once to get to the result.
So I did - results were not as good as I expected - It would take >15 builds to get just one image.Besides the fact that the sstate-cache is steadily growing, which limits build time even more, as
within the 50min the "fresh" cache needs to be packed and uploaded - very bad.
A few other pitfalls I recognized on the way

• logfile limit is 4MB at Travis CI, which is easily reached with a typical bitbake build.I worked around that by running "bitbake -q", which disables print out to console.
  Now you need to run each command with a "travis_wait" prefix, as Travis terminates any command which isn't printing out something at least every 10min

• As bitbake was run with "bitbake -q" there is no way to tell how much work has been performed within the given time.
  I worked around this by appending
  'echo "Tasks performed "$(grep -o "do_.*Succeeded" build/tmp/log/*/*.log | wc -l' to my build pipeline so I got 'Tasks performed 512' in the build console

Right back to start

Next I thought, let's do the minimal compliant solution and lets just check if bitbake is capable of parsing the setup of multiple layer-repo (incl. mine) - so I just added "bitbake -p" as the final step to my pipeline - First step successfully accomplished - I had the automated information from each
push/pull request if the result is still parsable.

First step done - Now for the real work...

Now I just needed to find a way of building the software at least weekly to check if it is all really working.Luckily I was hosting (and by the time of this writing, I still am) on GitHub and read the advertisement that they offering a CI-like service soon - which is available now, if you subscribe to a beta program - it's currently called Github Actions.
It offers (last update: 2019/10/22)

• 2 vCPUs
• 14 GB HDD
• 6h max. job run time

especially the last point sounded promising.The whole setup could be done over the GitHub UI via browser - and I did coded a basic nightly pipeline in a few minutes - and it worked!
After roughly 4.5h I had the result that my build was working with latest repo HEAD.
Finally I was more than happy, I had a fully functional CI without any costs, setup and (most important) without wasting days or weeks in hacking together some weird setup
(For some of you that have been doing stuff with ArchLinux AUR, so you might know what I'm talking about - pure PITA).

Final considerations

My personal setup

On each push / pull request

• Travis CI checks on any possible combination of layer (currently 4 in my case), if the result is still parsable by bitbake
• Result is presented within in GitHub at the corresponding commit
• A merge of PR is only possible if the Travis CI build is good
• Testing by Travis CI takes ~7-10 minutes in total

Twice a week

• GitHub Actions checks on any possible combination of layer (currently 4 in my case), if the result is buildable by bitbake
• Result is presented by mail
• Testing takes around 5h in total, so it's normally done over night

Real world code

If you're now keen on "real world" code, just head over to my static code analysis layer and have a look at 

• the Travis CI pipeline
• the nightly pipeline

or have a look at the build information of past builds at

• Travis CI

Unfortunately the GitHub-Action results aren't public, so you have to try that yourself.

If you have any questions, remarks or just want to have a chat, feel free to write a comment.

If you liked this post - be sure to read the follow-up