Upgrade policy

Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:

  1. Protecting our infra from folks who have access to Jenkins

    → Upgrading quarterly is sufficient.

  2. Protecting our infra from attacks against folks who have access to Jenkins

    For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).

    → We should stay on top of security advisories and react more quickly than "in less than 3 months".

  3. Protecting our infra from other 3rd-parties that affect Jenkins' security

    For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins orchestrator or workers permissions, i.e. root.

    → We should stay on top of security advisories and react more quickly than "in less than 3 months".

Upgrade steps

  • Preparation:

    • ☐ Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.
  • Deployment:

    • Take note of currently running builds before starting the upgrades.

    • Deploy Jenkins upgrade to latest version available using Puppet.

    • Generate a list of up-to-date plugins by running this Groovy script in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.

    • Generate updated Puppet code for tails::jenkins::master using this Python3 script and the output of the above script.

    • Deploy plugin upgrades using the code generated above.

    • Restart all agents.

    • Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):

      sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update
  • Wrap up:

    • Go through warnings in Jenkins interface.

    • Manually remove uneeded plugins from /var/lib/jenkins/plugins.

    • Restart builds that were interrupted by Jenkins restart.

    • Update the Jenkins upgrade steps documentation in case there were changes.

    • Schedule next update.

Agent to controller connections

These are the steps a Jenkins agent does when connecting to the controller:

  1. Fetch connection info from http://jenkins.dragon:8080 (see the tails::jenkins::slave Puppet class).

  2. Receive the connection URL ("Jenkins URL", manually configured in Configure System).

  3. Resolve to (because of libvirt config).

  4. Connect using HTTPS to

  5. Learn about port 42585 (fixed "TCP port for inbound agents", manually configured in Configure Global Security).

  6. Finally, connect using HTTP to

Generating jobs

We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.

The first brick extracts the list of active branches and output the needed information:

This list is parsed by the generate_tails_iso_jobs script run by a cronjob and deployed by our puppet-tails tails::jenkins::iso_jobs_generator manifest.

This script output YAML files compatible with jenkins-job-builder. It creates one project for each active branch, which in turn uses several JJB job templates to create jobs for each branch:

  • build_Tails_ISO_*
  • reproducibly_build_Tails_ISO_*
  • test_Tails_ISO_*

This changes are pushed to our jenkins-jobs git repo by the cronjob, and thanks to their automatic deployment in our tails::jenkins::master and tails::gitolite::hooks::jenkins_jobs manifests in our puppet-tails repo, these new changes are applied to our Jenkins instance.

Passing parameters through jobs

We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:

  • the Parameterized Trigger plugin, whenever it's sufficient

  • the EnvInject plugin, for more complex cases:

    • In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.

    • This file is used by the build job itself, to setup the variables it needs (currently only $NOTIFY_TO).

    • Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.


See automated builds in Jenkins.


See automated tests in Jenkins.