Automatically check your yocto-build for known security issues

Have you ever faced software security alerts in your embedded product or project? I did - several times. Mostly this had been going in the following way

• read an article or got a mail about the security issue
• checked (or may double-checked) the version actually used in the product is affected
• informed head of development (or head of security) that there is a problem in the software
• made several rounds of discussions and planning (which could take some time)
• included the fix, if there was one available
• did all the needed Q&A process
• deployed the fixed version
• kept the fingers crossed that software will be installed to most of the devices before anyone actually exploits the issue

The chain of disaster already starts at the first bullet - it's far too late - the whole process usually takes at least a few days - days devices in the field might be exploitable - and who's responsible in the end? Take a guess - mostly it's you.

Of course you can put yourself on the CVE-mailing-list and hope the best that you will find the time to read through all the findings that might affect your software - When a release is close I've my doubts that this will happen.

So there got to be a way of (at least semi-) automating this.
When you look around on the web you will surely find cve-check-tool which does check the version of your software package against known security issues.

If you have been into YOCTO/Open-embedded as a build-system there is also a wrapper around the cve-check-tool that comes in handy.
It does check the version of your bitbake-recipe and also includes some logic to filter out patches applied via patch-files to the recipe.

The sad thing is that this function present it's work in a NOT machine readable text form (as a file and on console).

Long story short I wasn't happy with that and I made a wrapper around the wrapper to create CI ready output files containing all the needed information.

You can find this as a part of my static code analysis layer

After this rather long preamble - how can one automate the cve-check?
What you need is

• CI like Jenkins
• static code analysis layer included into your layer construction

best idea would be to create a nightly job for your target build.

The following will show an example pipeline configuration using Jenkins, but it can easily adapted to any CI tool you are using.

If your using a pipeline configuration the actual build may look like this

stage "build"{
    steps {
        step{
             sh """
                 cd $WORKSPACE
                 . ./oe-build-init-env
                 bitbake my-target
             """
        }
    }
}

this can appended (or even replaced) by the following

stage "check-cve"{
    steps {
        step{
             sh """
                 cd $WORKSPACE
                 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SCA_ENABLE SCA_AUTO_INH_ON_RECIPE SCA_AUTO_INH_ON_IMAGE SCA_ENABLED_MODULES SCA_AUTO_LICENSE_FILTER"
                 export SCA_ENABLE="1"  ## globally enable sca
                 export SCA_AUTO_INH_ON_RECIPE="1" ## run on all recipes
                 export SCA_AUTO_INH_ON_IMAGE="0" ##  don't run on images
                 export SCA_ENABLED_MODULES="cve-check" ## use only the cve-check module
                 export SCA_AUTO_LICENSE_FILTER="" ## Don't care about the license of the package
                 . ./oe-build-init-env
                 bitbake my-target --runall=do_cve_check
             """
        }
    }
}

When this steps has run all the recipes that are actually used for your target build have been checked.

In your pipeline in the post process step insert the following

post {
      always {
              recordIssues tool: checkStyle(pattern: '**/sca/cve-check/checkstyle/*.xml'), qualityGates: [[threshold: 1, type: 'TOTAL', unstable: true]]      }
}

So now your build will be automatically failed or unstable if there is any open CVE-issue detected in your software (by the way only in the modules that are actually used for building your software). This can easily happen over night and when you get to the office in the morning you will have all the needed information condensed into a readable (and handleable) chunk of information.

That's it. One more thing that you don't have to focus on with your embedded product.

To be fair, this process does not cover all aspects of checking for security issues in your code base. It mainly just tracks down versions of packages and matches those with known information. So there is no protection to be expected against unknown or non-disclosure issues - but let your developer life become at least a little easier by automating things that could be automated.

Just a few words on the possible pitfalls:

• be sure to have access to online resources from your CI-building-node (cve-check-tool does get a fresh copy of it's database and therefore needs access)
• create appropriate mail-settings in the job, so that everyone important is getting the information in case of a finding
• The shown pipeline configuration does NOT build the software, it just performs the needed checks - so if you also want to build your software, you may include the needed configuration into your distribution configuration (see sca-layer-readme for details)