On 1 December 2021, we began receiving sporadic reports of delivery failures from proton.me addresses to Gmail. This corresponded with a dramatic decline in proton.me’s domain reputation as seen via Gmail Postmaster Tools and an increase in sending from known bad IP addresses.
It was clear both from the bad sending IPs (mostly in Russia) and our own metrics that the spam emails damaging Proton Mail’s domain reputation were not coming from our servers. However, the Postmaster Tools indicated that all emails being received by Gmail from proton.me were “fully authenticated”, including the fraudulent ones.
This, in turn, caused the fraudulent emails to feed into Google’s algorithm for determining domain reputation and lowered it enough that the deliverability of legitimate emails from our servers was affected as well.
We suspected a DKIM replay attack, where a single spam email originally sent from Proton Mail was being resent to many Gmail users in an attempt to exploit our deliverability and reputation to get around Google’s anti-spam measures. At one point, roughly 98% of the emails Gmail received that claimed to be from Proton Mail were actually spam, meaning the spammers were sending an amount of emails that was equivalent to 50 times our normal outgoing traffic to Google.
We immediately rotated our DKIM signing key to (temporarily) prevent the emails from passing DKIM and contacted Google’s counter-abuse team, who quickly implemented a fix to Gmail’s spam filters and restored legitimate email delivery.
Emails sent from proton.me, proton.me/mail, and custom domains were not affected by this issue.
How DKIM replay attacks work
Before we can explain how this happened, we first need to outline how emails are structured, delivered, and authenticated on the internet.
Emails are MIME (Multipurpose Internet Mail Extensions) messages, consisting of headers and sections that contain the message body and possibly attachments. The headers contain some fields that will be familiar to any email user (To, CC, From, Subject) but also hidden information that is used to authenticate the email.
However, none of the headers are actually used to route the email to its final destination. The recipient and sender of the email are specified separately as part of the email envelope, which is a very suitable metaphor. If the email message, including its headers, is equivalent to a paper letter, then the email envelope in which the “letter” is placed contains the recipient and return addresses, like a real envelope.
The key point is that the recipient on the envelope does not have to match any recipient in the To or CC headers — typical examples of this are emails sent via BCC or addressed to “undisclosed recipients”.
Perhaps even more surprisingly, the envelope sender does not have to match the email From header. This also has a legitimate purpose — mass mailings often specify a different “return address” than the From header to analyze deliverability problems or use third-party services to send emails. This flexibility is also important for enabling users to forward messages from one mailbox to another.
These return addresses are authenticated via SPF (Sender Policy Framework), which authorizes the sending of messages by specific servers or IP addresses using special DNS records. But this only validates the server sending the email; it does not ensure that the email’s content has not been tampered with. For that, we need DKIM (Domain Keys Identified Mail).
DKIM also uses special DNS records, but rather than a list of IP addresses, these records contain keys used to sign the email content and certain associated headers. The resulting cryptographic signature is appended to the message as a special header, and the recipient mail server or client can verify this signature against the email content on delivery. But the DKIM domain in the signature can also be different from the one in the From header in the message itself. If the DKIM signature is verified, this only confirms that the message went through the signing domain’s mail servers and has not been modified since, not that the message originated from where it claims.
To actually verify that the domain in the From header sent the message, we need DMARC (Domain-based Message Authentication, Reporting, and Conformance). To pass DMARC, an email must pass either SPF or DKIM, and the domain in the From header must be “aligned” with the corresponding SPF or DKIM domain. The From header is what the user ultimately sees, so DMARC is a critical part of ensuring that an email originated from where it claims it did as it is the only policy that connects the From header to either the sending or signing domain.
(ventana nueva)Now let’s return to the attack. The reason that DKIM replay attacks work and why Gmail considered those replayed spam mails “authenticated” comes down to the fact that DMARC requires DKIM or SPF alignment, but not both. The replayed message itself had a valid DKIM signature from proton.me, which meant it passed DMARC. This message was then sent enough times that it influenced proton.me’s domain reputation in Gmail’s system, eventually becoming low enough to affect deliverability for legitimate email.
How we prevent replay attacks
The fact that DMARC passes if either the DKIM domain or the SPF domain align with the From domain is a feature of the specification, not a bug. In particular, it enables email forwarding and allows email sending by trusted third parties. However, for email service providers, such as Proton, Gmail, or Yahoo, it also enables these kinds of replay attacks, as any user can send an email, get it signed by the corresponding domain, then resend it with the signature intact.
This is one reason that Proton and other email providers invest heavily in their own anti-spam technology and systems. These systems are complex and often rely on complicated heuristics to separate spam from legitimate email. In this case, the attackers found a vulnerability in Gmail’s anti-spam system and were able to exploit it.
We appreciate Google’s responsiveness in addressing the issue.
How you can prevent domain impersonation on your domain
DKIM replay attacks are an issue primarily for email service providers or other organizations that offer email addresses on a shared domain. However, email authentication attacks in general are a risk for any organization. Here are some tips to help you ensure no one else can impersonate your domain or use it to send fraudulent messages.
- Set up SPF, DKIM, and DMARC – Although imperfect, these policies are key to ensuring that your emails are delivered and resistant to being spoofed. If you use Proton to host your domain, our domain setup wizard will explain how to set it up and protect it from being spoofed.
- Rotate your DKIM keys regularly – Rotating our DKIM keys allowed us to quickly stop the attack and buy time for the permanent solution. Although tedious and risky to do manually, Proton’s DKIM key management system(ventana nueva) allowed us to easily do this in minutes, and this system is the same system used for all domains hosted at Proton. The system also automatically rotates keys regularly to reduce the risk of key compromise.
- Oversign From, To, and CC headers – Most DKIM implementations always sign the From, To, and CC headers if they are present in an email, preventing them from being modified if the message is resent. However, if these headers are missing, they are often unsigned, opening the door to replay attacks with forged headers that make the fraudulent emails seem legitimate. Oversigning mitigates these attacks by signing these sensitive headers in all cases, even if they are blank. If you use Proton to send your email, this oversigning is done for you automatically by our mail servers.
- Be careful of subdomains – If you use CNAME records to delegate parts of your website to third parties, you may be also allowing these third parties to send email on behalf of your main domain. This is because, by default, DMARC considers domains aligned if they have a parent-child relationship — that is, sub.example.com aligns with example.com. You can force exact match alignment for SPF, DKIM, or both with the aspf and adkim options in your DMARC policy. However, be aware that this can affect third-party mailing service integrations.
A lot happens after you click Send in Proton Mail
If we’re doing everything correctly on our end, email delivery seems as easy as a single click, but the truth is that it relies on a complex web of interlocking, interdependent policies. Events like this replay attack demonstrate just how complicated it can be to verify something as seemingly simple as “Who sent this email?”
At Proton Mail, we are constantly investigating new protocols and policies to ensure that the millions of emails sent using our platform every day are delivered reliably and securely. We also continuously monitor incoming mail to ensure our authentication checking has been optimized and we have systems and analysts in place 24/7 to mitigate spam and phishing attacks.
This is a vital part of creating an internet where privacy is the default, and we could not do it without the support of the Proton community. Thank you.
