1. Changes
  2. Reproducibility
  3. Automated test suite
  4. Tor Browser
  5. Thunderbird
  6. Tor
  7. WhisperBack
  8. Root access control
  9. Virtualization support
  10. APT (automate: #8164)
  11. Incremental upgrades
  12. Unsafe Web Browser
  13. Tails Verification
  14. Real (non-VM) hardware
  15. Documentation
  16. Internationalization
  17. Misc

Some test results that might be useful to keep are saved.

Read this document from the branch used to prepare the release.

Changes

Keeping an eye on the changes between released versions is one of the many safeguards against releasing crap.

Source

Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the tails-version command.

Then, from the source tree, see the diff:

git diff --find-renames <old ISO commit>..<ISO commit>

e.g. git diff --find-renames 334e1c485a3a79be9fff899d4dc9d2db89cdc9e1..cfbde80925fdd0af008f10bc90c8a91a578c58e3

Result

Compare the list of bundled packages and versions with the one shipped last time. .packages are usually attached to the email announcing the ISO is ready.

/usr/bin/diff -u \
    wiki/src/torrents/files/tails-amd64-3.1.packages \
    tails-amd64-3.2.packages \
    | wdiff --diff-input  --terminal

Check the output for:

  • new packages that may cause harm or make the images unnecessarily big
  • packages that could be erroneously removed
  • new versions of software we might not have audited yet (including: does the combination of our configuration with software X version Y+1 achieve the same wished results as with software X version Y?)

Image size

Check the image size has not changed much since the last release.

In a directory with many Tails ISO images:

find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 5

Reproducibility

This section can not be done by the RM.

  1. Download the ISO and all the IUKs that upgrade to the version you are testing.

  2. Clear-sign the hashes of all products using your OpenPGP key:

     sha512sum tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso \
     | gpg --clear-sign

  3. Send the output of the previous command to the Trusted Reproducer, whose name is on the release calendar.

Automated test suite

Our long term goal is to eliminate the manual test suite (except the parts which require real hardware) and have the automated test suite run all our tests. It's design, and how to write new tests, are documented on a dedicated page.

Running the automated test suite

See setup and usage.

Do point --old-iso to the ISO of the previous stable release.

Automated test suite migration progress

The manual test suite below either contains tests that cannot be automated, has no automated test implemented yet, or has a test implemented, but it either hasn't been reviewed, had a confirmed pass by someone other than the test author, or has issues. The latter is tracked by tickets prefixed with todo/test_suite:.

Tor Browser

Miscellaneous functionality

  • Test if uBlock works:
    • The uBlock icon must be visible.
    • Visit a website that normally displays ads, such is https://www.nytimes.com/. The ads should not be displayed and the uBlock icon should display a strictly positive number of blocked elements.

Security and fingerprinting

  • Run the tests the Tor Browser folks use and compare to the last released version of Tails. Results should not be worse. (automate: #10260)
    • For the "evercookie" test to work, you may have to disable uBlock on its web page.
  • Compare the fingerprint Tor Browser in Tails with the fingerprint of the same version of Tor Browser (running on Linux outside of Tails), using at least https://panopticlick.eff.org/ (automate: #10262). Click "Show full results for fingerprinting" to see the details we're interested in.
    • The exposed User-Agent should match the latest Tor Browser's one.
    • Ignore the result of the "blocking tracking ads" and "blocking invisible trackers" tests, which seem unreliable (we've seen different results for the very same version of Tor Browser).
    • Update the Browser fingerprint section of the known issues if needed.
  • WebRTC should be disabled: (automate: #10264)
    • In about:config check that media.peerconnection.enabled is set to false.
    • http://mozilla.github.io/webrtc-landing/, especially the getUserMedia test. It's expected that the audio test works if you agree to share a microphone with the remote website; anything else should fail.
    • http://net.ipcalf.com/ should display ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1
  • Running /usr/local/lib/getTorBrowserUserAgent should produce the User-Agent set by the installed version of Torbutton, and used in the Tor Browser. (automate: #10268)

Thunderbird

  • Check mail over IMAP using:
    • a hidden service IMAP server (e.g. Riseup, zsolxunfmbfuq7wf.onion on port 993 with SSL).
  • Check mail over POP using:
    • a hidden service POP server (see above, on port 995 with SSL).

  • Send an email using:

    • a hidden service SMTP server (see above, on port 465 with SSL).

  • Check that the profile works and is torified:

    1. Send an email using Thunderbird and a non-anonymizing SMTP relay (a SMTP relay that writes the IP address of the client it is relaying email for in the Received header).
    2. Then check that email's headers once received, especially the Received: one.
  • Also check that the EHLO/HELO SMTP message is not leaking anything at the application level:
    1. Start Thunderbird using the GNOME Applications menu.
    2. Disable SSL/TLS for SMTP in Thunderbird (so take precautions for not leaking your password in plaintext by either changing it temporarily or using a disposable account). Or better, configure StartTLS, since it will send two EHLO/HELO: one before TLS is initiated; one after. The assumption here is that Thunderbird will send the same both times.
    3. Run sudo tcpdump -n -i lo -w dump while sending an email to capture the packets before Tor encrypts it, then close tcpdump. Note that the packet containing EHLO/HELO will be sent really early, so even if the email failed (e.g. because the mail server doesn't support plaintext SMTP on port 587) we are ok.
    4. Check the dump for the HELO/EHLO message and verify that it only contains 127.0.0.1: sudo tcpdump -A -r dump | grep EHLO

Tor

WhisperBack

  • I should be able to send a bug report with WhisperBack.
  • When we receive this bug report on the tails-bugs mailing list, Schleuder tells us that it was sent encrypted.

Root access control

  • Check you cannot login as root with su neither with the amnesia password nor with the live one. (automate: #10274)

Virtualization support

  • Test that Tails starts and the browser launches in VirtualBox.

APT (automate: #8164)

 grep -r jenw7xbd6tf7vfhp.onion /etc/apt/sources.list*
  • Make sure the Tails repository suite in matching the release tag (for example the release version number) is in APT sources.
  • Make sure the Tails repository unversioned suites (e.g. testing, stable and devel) are not in APT sources.

Incremental upgrades

  • List the versions from which an upgrade path to this one is described. On the master branch:

    git grep -E -l "  version: '?0.23'?" \
    wiki/src/upgrade/v1/Tails/*/*/test/upgrades.yml

    … replacing "0.23" with the version you are testing.

  • For each description file, open it and verify if it allows incremental upgrade or only full upgrade.

  • For each previous version from which an upgrade path is described, install it and try to upgrade:

    • For every incremental upgrade path: make sure the resulting updated system "works fine" (boots, pretends to be the correct version, and the following components work fine: Tor, Tor Browser, Unsafe Browser).
    • For upgrade paths that only propose a full upgrade: make sure the user is told to do a manual upgrade.

    If:

    then:

      echo 'TAILS_CHANNEL="test"' | sudo tee --append /etc/os-release && \
  tails-upgrade-frontend-wrapper

    Else, use a local test setup:

    • A web server on the LAN.
    • A copy of wiki/src/upgrade from the stable or testing branch, for example in /var/www/tails/upgrade/v1/Tails/3.14~rc2/amd64/stable/updates.yml

    • A copy of the iuk directory of our HTTP mirrors, for example in /var/www/tails/stable/iuk/Tails_amd64_3.14-rc2_to_3.14.iuk.

      To synchronize your local copy:

      torsocks rsync -rt --progress --delete mirrors.rsync.tails.boum.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/

    • Patch /etc/hosts in Tails to point to your web server:

      echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts

    • Patch sudo configuration to allow passing arbitrary arguments to tails-upgrade-frontend:

      sudo sed -i \
    -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
    /etc/sudoers.d/zzz_upgrade

    • Call the upgrader must be called, from inside the system to upgrade, with every needed option to use the local web server rather than the online one, for example:

      DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
tails-upgrade-frontend-wrapper --override-baseurl \
http://192.168.1.4/tails

Unsafe Web Browser

  • Browsing (by IP) a FTP server on the LAN should be possible. (automate: #10252)

Tails Verification

  • The goal is is to check that Tails Verification works in Tor Browser in the version of Tails we are testing here. Tails Verification only supports verifying the current release so for example, when doing tests for the Tails 3.9 release, we use it in the tentative Tails 3.9 to verify the Tails 3.8 ISO image.

    1. Start the Tails that you are testing.

    2. Start Tor Browser.

    3. Visit https://tails.boum.org/install/download/.

    4. If you already have the ISO image for the last, already released version of Tails, i.e. the one that's advertised in the sidebar of our production website, then click "I already downloaded Tails $N". Otherwise follow the instructions on that web page to download the ISO image.

    5. Install Tails Verification.

    6. Click the Verify Tails $N button.

    7. The verification should be successful.

Real (non-VM) hardware

[can't-automate]

  • Boot on bare-metal from USB. Measure the boot time (from the syslinux menu until the GNOME desktop is ready -- quickly press ENTER in the Greeter) and compare with the boot time of the previous Tails version. The new one should not be significantly slower to start.
  • Boot on bare-metal from DVD. Measure the boot time (from the syslinux menu until the GNOME desktop is ready -- quickly press ENTER in the Greeter) and compare with the boot time of the previous Tails version. The new one should not be significantly slower to start (for release candidates we do not always update the squashfs sort file, so then it might be ok if somewhat slower).

Documentation

  • The "Tails documentation" desktop launcher should open the doc page (automate: #8788):
    • in one language to which the website is translated
    • in one language to which the website is not translated (=> English)
  • Browse around in the documentation shipped in the image. Internal links should be fine. (automate: #10254)

Internationalization

Boot and check basic functionality is working for these (language, region) tuples:

  • Arabic - Egypt
  • Chinese - China
  • Deutsch - Deutschland
  • English - USA
  • Español - España
  • Français - France
  • Italiano - Italia
  • Persian - Iran
  • Português - Brasil
  • Russian - Russian Federation
  • Tiếng Việt - Vietnam

You really have to reboot between each language.

  • The chosen keyboard layout must be applied. (automate: #10261)
  • The screen keyboard must (automate: #10263):
    • work in Tor Browser when activated after the browser has started;
    • work in Thunderbird when activated after Thunderbird has started;
    • be auto-configured to use the same keyboard layout as the X session (known exceptions: Chinese, Persian, Português, Russian, Tiếng Việt).
  • In the Tor Browser:
    • DuckDuckGo must be the default, pre-selected search plugin. (automate: #10265)

Spellchecking

To see which among the supported locales there should be no spellchecker, run this in a Tails Git checkout of the commit the release under testing was built from:

git grep NO_SPELLCHECKER_LOCALES= config/chroot_local-hooks/11-localize_browser

Then do the follow in the same Tor Browser session running in the en_US.UTF-8 locale (or whatever locale you are most comfortable identifying other language names in):

  • Check that the expected languages are listed in the list of languages for spell checking. (automate: #10269)
  • For a few languages, check the spell checking:
    • Type something in the textarea.
    • Right-click and select a language.
    • Verify that the spelling suggestion are from that language. (automate: #10271)

Misc

  • Check that all seems well during init: (automate: #10277)
    • systemctl --failed --all should say 0 loaded units listed
    • the output of journalctl should seem OK.