Thunderbird Blog
By Justus and Neal | April 8, 2021
We are thrilled to release the first version of the Octopus, analternate OpenPGP backend for Thunderbird built on top of Sequoia.
The Octopus is a drop-in replacement for RNP, the OpenPGP libraryshipped with Thunderbird 78. In addition to providing all of the RNPfunctionality that Thunderbird uses, the Octopus also includes anumber of enhancements. These fall into several categories. TheOctopus restores some functionality that was present in Enigmail, butremoved or has not yet been reimplemented in Thunderbird’s OpenPGPintegration. In particular, the Octopus uses GnuPG’s keystore,interacts with gpg-agent, integrates GnuPG’s web of trust information,and updates certificates in the background. The Octopus includes anumber of security fixes and improvements. For instance, it fixesThunderbird’s insecure message composition, and automatically encryptsin-memory secret key material at rest. The Octopus adds a fewperformance improvements, such as, parsing the keyring in thebackground and using multiple threads. And, the Octopus has bettersupport for parsing less usual, but not necessarily esoteric,certificates and keys.
Its name and design reflect the mythical thunderbird, a Native American symbol of power and strength, with three tail feathers representing past, present, and future. Made by hand from the finest suede or leather, with colorful beading and cotton laces, these comfy women’s moccasins are a unique spin on a classic boat shoe. Through this blog, we shared the manual method that will help you to convert Thunderbird mail to Outlook PST. And if you want a quick and accurate result you can use SysInspire MBOX to PST Converter tool. This software is best in its functionality and also gives a free trial version for the user’s satisfaction. Feeds are often used by blogs, but more traditional websites can make any content available in this format. When you subscribe to a site's feed link, your feed reader (for example, Thunderbird) automatically checks the location and downloads the articles to local folders. Similar Blog Post – Convert IncrediMail to Outlook. Client Testimonials. I was looking for best software for IncrediMail to Thunderbird conversion. My friend suggested this BitRecover Software. Also, I have used suggested software to switch email client from IncrediMail 2.5 to Thunderbird inclusive of all email items.
The Thunderbird product is developed and managed by the Thunderbird Project, which is collectively the community (both paid staff and volunteers) and the Thunderbird Council: The community has continuously shipped releases with bug fixes and enhancements for many years. Read more at the Thunderbird Blog.
The update from Thunderbird 68 to Thunderbird 78 brought a big change:the old extension mechanism was retired in favor of a safer approach,WebExtensions. This was an invasive change. Many add-ons includingthe popular Enigmail extension and the Autocrypt extension had tobe reworked or even rewritten.
Recognizing that OpenPGP is essential for many existing Thunderbirdusers, the Thunderbird project decided to natively support OpenPGP.This is a positive development as it makes OpenPGP accessible to morepeople, and allows for tighter integration between the OpenPGP supportand Thunderbird. In September 2020, Thunderbird 78.2.1 was releasedand, for the first time, Thunderbird offered OpenPGP functionality bydefault.
Rather than starting from scratch, the Thunderbird team lifted a lotof the user interface and plumbing code from Enigmail. They alsoreplaced GnuPG with RNP, which is significantly easier to bundlethan GnuPG.
Unfortunately, due to time constraints many features that Enigmailusers enjoyed were removed to ensure that Thunderbird users at leasthad something that understands rudimentary OpenPGP. For instance,instead of using OpenPGP’s native authentication mechanisms,Thunderbird has a custom acceptance mechanism. Because the usualOpenPGP artifacts are not created, it is difficult to synchronizethese judgments with another OpenPGP implementation or even anotherThunderbird installation.
Our new project, the Octopus, is an alternative OpenPGP backend forThunderbird. The Octopus implements the RNP functionality thatThunderbird uses in an ABI compatible manner to RNP, i.e. it is adrop-in replacement for the library ships with Thunderbird 78. Inaddition to implementing the functionality that RNP provides in termsof Sequoia, it also restores missing features, like GnuPGintegration, and has a number of non-functional advantages includingsecurity fixes and additional protections.
In this demo, you can see Aron Salih sending an encrypted mail toElise Sophia using the stock Thunderbird. Then, Elise Sophiadownloads Thunderbird, replaces the RNP library with the Octopus,decrypts the mail, and replies.
We recently learned (1, 2) that Red Hat decided to disableOpenPGP support in their Thunderbird builds, because RNP uses Botanas the underlying cryptographic library, but Botan is not supported byRed Hat Enterprise Linux (see Fedora Crypto Consolidation).
A tune for mac. Seeing that Sequoia’s default cryptographic library, Nettle, issupported by Redhat Enterprise Linux, we realized that we could fillthe gap by providing a library that uses Sequoia and provides the sameinterface that RNP provides, or at least the subset that is actuallyused by Thunderbird.
Later we realized that it is not complicated to restore some of theoft-requested functionality directly in the Octopus. And, as we alsomissed that functionality ourselves, we decided to add it to theOctopus.
This is the power of free software: Freedom #1 allows us to modify aprogram freely to our needs, and freedom #3 allows Red Hat to ship themodified program for the benefit of their users.
The Octopus implements all of the functions that Thunderbird uses.And, we are happy to report, all of Thunderbird’s OpenPGP tests passwith flying colors. You can read the full log here, but the gistis:
And, here is the video of the Mochitests running. It is too fast tosee any details, but it is still fun to look at:
You can find build instructions in the projects README file. Thatfile also includes instructions on how to use our precompiled binariesfor Windows.
The Octopus is actively developed by the Sequoia PGP team. We areprimarily financed by the pep foundation. Our mandate isspecifically to improve the OpenPGP ecosystem, and more generally toimprove internet freedom tools.
Currently, the Octopus is developed outside of Thunderbird. There aretwo main reasons for this. First, the Thunderbird developers do notwant to invest resources in supporting a new OpenPGP backend afterhaving recently invested in RNP. Second, Sequoia and the Octopus arelicensed under the GPLv2+. Although the GPLv2+ is compatible withMPL 2.0, adding GPLv2+ code to Thunderbird is not currently alignedwith their licensing strategy.
Admittedly, implementing another library’s API is a bit dodgy.Long-term we’d like to see Thunderbird adopt SOP. SOP is aStateless OpenPGP API. The current version only targets the CLI, butthere is work to standardize a C API and ABI. Using SOP would allowincreased choice for users. It also gives Thunderbird more freedom toeasily change their implementation should the need arise.
The Octopus includes a number of additional features that enhanceThunderbird. We suspect that for many users, the integration withGnuPG will be most interesting. But, we’ve also added a Parcimonieimplementation, and fixed some security weaknesses.
GnuPG Keyring Integration
When Thunderbird starts, it asks RNP to parse its keyring. At thispoint, the Octopus also runs gpg --export and includes that in theresults. This makes the user’s GnuPG keyring available toThunderbird. (From that point on the Octopus also monitors GnuPG’skeystore for updates.) This can be disabled by setting theGNUPGHOME environment variable to /dev/null as follows:
The certificates imported from GnuPG can be used as normal. It ispossible to examine them in Thunderbird’s OpenPGP Key Manager, settheir “acceptance”, etc.
There are two known limitations. First, the first time an OpenPGPoperation is performed, Thunderbird scans the keystore and creates anindex of the available keys. Thunderbird does not currently updatethis index on its own. Thus, keys that are added to gpg’s keystorewill not be visible to Thunderbird until either the cache is manuallyflushed (OpenPGP Keyring Manager, File, Reload Key Cache), oryou restart Thunderbird. Second, if you remove a certificate managedby GnuPG using the Thunderbird certificate manager, it will be removedfrom the in-memory keystore, but it is currently not actually removedfrom your gpg keystore. As such, it will reappear the next timeThunderbird loads the keyring.
The Octopus carefully keeps track of what certificates were loadedfrom GnuPG and only writes them out to Thunderbird’s keyring if theyhave been modified; modified certificates are not currently writtenback to GnuPG. But, closer integration with GnuPG’s keyring isplanned.
gpg agent Integration
The Octopus automatically monitors what keys are loaded into gpg’sagent, and reports to Thunderbird that secret key material isavailable for them. This means that it is trivial to mark a keymanaged by the agent as a personal key in Thunderbird withoutmodifying Thunderbird’s configuration files. Also, attempts to decryptmessages encrypted to a key managed by the agent are automaticallyforwarded to the agent for decryption.
Unlike Thunderbird the Octopus talks directly to the agent. Thus,it is not necessary to install GPGME; you only need to have gpg inyour PATH.
GnuPG’s Web of Trust Data
Thunderbird only supports a custom “acceptance” mechanism forauthenticating OpenPGP certificate. Thunderbird ignores keysignatures, and it is not possible to add certification authorities.
When Thunderbird starts up, the Octopus reads gpg’s trust database,and merges it into Thunderbird’s acceptance database. This meanscertificates that are considered authenticated by GnuPG are alsoconsidered authenticated by Thunderbird.
This integration is done carefully. If a user has manually accepted acertificate in Thunderbird, that setting is not overridden. Thishappens not only during the initial import, but also later: when theuser accepts a certificate in Thunderbird, the Octopus detects thisand will no longer update that certificate’s acceptance based onGnuPG’s trust database.
The Octopus monitors GnuPG for changes to its trust database. So,unlike when a new certificate is added to GnuPG’s keystore, it is notnecessary to restart Thunderbird to notice changes to the trustdatabase.
Parcimonie
Parcimonie is a feature that automatically refreshes the user’sOpenPGP certificates in the background using a number of privacypreserving techniques. In particular, updates are staggered, and thetime between updates is drawn from a memoryless distribution tofrustrate an attacker who wants to predict when a user will check foran update. Enigmail had its own version of this mechanism, but it wasremoved when Enigmail was integrated into Thunderbird.
The Parcimonie feature in the Octopus currently checks for updates onkeys.openpgp.org and in the appropriate Web Key Directories (WKDs)using the aforementioned privacy preserving mechanisms. It checks forupdates for all non-revoked, valid certificates about once a week, onaverage. It also supports merging updates from User ID-lesscertificates.
Before importing a certificate, we first check if it appears to beflooded. If so, we strip third-party certifications from keys thatwe don’t have a certificate for as those certifications areeffectively useless.
This feature is controlled by the net feature, which is enabled bydefault. To disable it (and elide the dependencies on sequoia-netand tokio), build as follows:
Weak Cryptography
The Octopus uses Sequoia, which rejects cryptographic algorithms thatare known to be weak by default. Unfortunately, RNP still acceptsMD5, among other vulnerable algorithms, without warning. Thunderbirdhas patched MD5 out of the version of RNP that they distribute,however, Thunderbird continues to support unlimited use of SHA-1,which is known to be vulnerable to collision attacks. Sequoia, andby extension, the Octopus, rejects certificates and messages that useweak cryptographic primitives. Because RNP does not have a mechanismto indicate that a certificate or component should not be used, theOctopus reports these keys as having expired one second after theircreation time.
Protection from Surreptitious Forwarding
When Thunderbird creates an email it uses the RFC 1847 Encapsulationmethod construct, which has been known to be broken for over 20years. The issue is that if Alice signs a message and sends it toBob, Bob can use the signature in a different context.
This security issue can be fixed by using OpenPGP’s intended recipientfeature. Then, if Bob forwards the signed message (e.g., “I owe you100 Euros”) to Carol, Carol’s MUA will mark the signature as invalidas she is not the intended recipient of the message.
In the Octopus, we detect the use of the encapsulation method andautomatically and transparently replace it with the safer combinedmethod whenever possible. Specifically, when we encrypt a message,we check whether we just generated the signed part. If so, we fix it.Because this generates another signature, if you are using a keymanaged by gpg’s agent, you may be prompted to authorize a secondsignature.
Sequoia also has a number of non-functional advantages relative toRNP.
Keys Encrypted at Rest
Sequoia automatically encrypts unencrypted secret key material inmemory when it is not in use. This makes secret key exfiltration alaHeartbleed much harder, and protects against Spectre, Rowhammer,etc.-style attacks. OpenSSH uses the same type of protection.
RNP has the concept of locking and unlocking keys, but this isexplicit, and Thunderbird does not always relock keys after use.
2003 Ford Thunderbird Forum
SHA-1 Mitigations
SHA-1 is broken. Unfortunately, SHA-1 is still widely used. Todeal with this Sequoia implements a number of countermeasures:
Sequoia uses SHA1-CD, a variant of SHA-1 that detects andmitigates collision attacks. This protection is also used byGitHub, among others.
Sequoia only accepts SHA-1 in safer contexts. For instance,SHA-1 over messages, and its use in third-party certifications arerejected by default. But, SHA-1 self-signatures that are notsuspicious are allowed.
Sequoia has announced a timeline to completely deprecate theuse of SHA-1: in 2023 SHA-1 will no longer be accepted fordigital signatures.
RNP accepts SHA-1 everywhere without any additional protections. Bydefault, it even acceptsMD5. Happily, Thunderbird carries apatch to disable MD5 support.
Collision Protection
Mac mail for yahoo mail. Sequoia includes a salt in signatures and self-signatures to defendagainst collision attacks, among others. OpenSSH does the samething. Should the collision resistance of another hash be broken,this will frustrate attackers trying to perform a Shambles-styleattack.
No Split Brain Problem
RNP maintains separate public and secret keyrings. This can lead to aso-called split-brain problem where a certificate is present in bothkeyrings, and confusingly one version is returned sometimes and theother version other times. This is also the model that GnuPG 1.xused, and is one of the reasons that GnuPG migrated to a singleOpenPGP keystore in GnuPG 2.0 with only the secret key material heldby the agent.
To avoid this problem, the Octopus merges the two databases. Toremain backwards compatible with RNP, when the Octopus writes out thecertificates, certificates with secret key material are written tosecring.gpg and those without are written to pubring.gpg.
Multi-threading
Thanks to Rust’s safer concurrency paradigms, it is less dangerousand less complicated for the Octopus to use threads than librarieswritten in other languages. The Octopus uses this, for instance, toparse keyrings faster. And to perform updates in the background.
OpenPGP Conformance
Sequoia implements nearly allof the OpenPGP RFC. The onlynotable missing bit is the lack of ElGamal support.
RNP doesn’t implement a number of important parts. For instance, itdoes not reject unknown critical subpackets and notations. This isa security problem. RNP doesn’t handle unknown packet versions.This is a future compatibility problem. RNP also doesn’t handle“esoteric” keys, like shared keys where only the encryption subkey’ssecret key material is shared. This is a compatibility problem. Moreexamples can be found in the OpenPGP interoperability test suite.
Over the past couple of months, we had several productive emailexchanges and a meeting with Magnus and Kai from the Thunderbird team.We’re grateful for the help they provided and their feedback.
This article explains how Thunderbird stores messages on the local disk drive and why it is necessary to compact messages periodically.
Table of Contents
Thunderbird stores messages using the MBOX file format. With this file format, all the messages in each of Thunderbird's folders are concatenated and stored as plain text in a single file on the local hard drive (located into the Mail and ImapMail directories of your Profiles folder).
As messages are added to a folder, the file containing the folder grows larger on the disk. However, when you delete a message or move it from one folder to another, the file on the disk does not automatically get smaller. This is because the original message is simply marked for deletion and hidden from view. It is not physically removed until you 'compact' the folder. This temporarily improves performance in large folders but, in time, the large file is less efficient to work with. Therefore, in order to reclaim disk space and improve Thunderbird's performance, folders must be 'compacted' periodically.
To compact a folder, Thunderbird opens the existing MBOX file on the disk (for example, the Inbox). Based on the rules for the MBOX mail format it reads the file one message at a time.
- If the message is still current, it is copied to a new temporary MBOX file, called Nstmp.
- If the message is marked as deleted or moved, Thunderbird skips that message and moves on to the next message.
This process is repeated one message at a time until the end of the file is reached. After that the original message storage file is deleted and the new one replaces it. This is followed by the generation of a new index for this message file (for example, called Inbox.msf).
The compaction process is done automatically in Thunderbird (since version 5) when it saves more than 20 MB of space on the disk.
You can also launch a manual compaction request if needed:
- To compact a single folder, right-click on the folder and select Compact.
- To compact all folders, select File > Compact Folders.
During a compaction process, progress is displayed in the Status Bar:
In the Thunderbird | PreferencesTools | OptionsEdit | Preferences | Advanced | Network & Disk Space menu, you can:
Thunderbird Resort Rizal Blog
- disable automatic compaction (which is not recommended because your folders won't be regularly cleaned of deleted messages)
- change the compaction threshold (for instance if you deal with a large volume of messages, you may use a larger threshold)
Camp Thunderbird Blog
- The Thunderbird Tweaks blog has a related article called 'Compacting :- What is it and Why must I do it.'