Skip to content

The DigiNotar and Symantec CA failures: how trust gets revoked

· 21 min read
Copyright: MIT
history pki web-security
Wordmark reading DISTRUST with a struck-through certificate authority root

Every padlock in your address bar rests on a short list of organizations a browser has decided to believe. A certificate authority signs a statement that says, in effect, this public key really does belong to the operator of google.com, and your browser accepts that statement because it ships with the CA’s root key baked into a trust store. The whole arrangement works only as long as the CA does its one job. So what happens when one of them doesn’t? Not a bug, not a phishing page, but the signing key itself producing valid certificates for domains the holder never controlled.

Two answers to that question shaped the modern web PKI. In 2011 a Dutch CA called DigiNotar was breached, an attacker minted a working certificate for *.google.com, and within weeks the company was bankrupt and erased from every trust store on earth. Six years later Symantec, then the largest CA in the business, was found to have mis-issued certificates at scale, fought the finding, and was slowly and deliberately distrusted by Google over the course of a year. One was a death by intrusion, the other a death by process. Together they define the two ways a CA loses the only thing it sells.

This post walks both cases as primary-source case studies. First DigiNotar: the breach, the fraudulent Google certificate, the Iranian man-in-the-middle campaign, and the speed of the collapse. Then Symantec: the slow accretion of mis-issuance findings from 2015 to 2017, the fight on a Google mailing list, and the staged distrust schedule Chrome actually shipped. The closing section pulls out what the two have in common, which is the mechanism by which a browser revokes trust, and why that mechanism is the real backstop of the system. For the structural background on how the trust store itself works, the companion piece on the web PKI trust store is the place to start.

How a CA earns trust in the first place

Before either failure makes sense, it helps to be precise about what trust means here. A root CA certificate is a self-signed public key that browser and OS vendors choose to embed in their products. Mozilla ships a root store with Firefox, Microsoft ships one with Windows, Apple ships one with macOS and iOS, and Google ships its own with Chrome on most platforms. Inclusion is a privilege granted after audits, and it can be withdrawn. When a vendor withdraws it, every certificate that chains up to that root stops validating, everywhere that vendor’s software runs, on the next update.

That last clause is the crux. There is no central kill switch for a CA. Revocation lists and OCSP exist for individual leaf certificates, but they are slow, often soft-failing, and easy to bypass for an attacker who controls the network path. The reliable lever is the trust store: remove the root, push an update, and the CA is gone. The cost is that the update has to reach users, and any certificate the CA legitimately issued breaks at the same moment as the fraudulent ones. That tension between collateral damage and decisive action runs through both stories.

It is worth sitting with why per-certificate revocation is not enough, because the DigiNotar case turns on it. The two classic mechanisms are the Certificate Revocation List, a signed file the CA publishes listing serial numbers it has revoked, and the Online Certificate Status Protocol, where a client asks the CA’s responder whether a given certificate is still good. Both have the same fatal property in an adversarial setting: they fail open. If the client cannot reach the CRL distribution point or the OCSP responder, most browsers historically treated the certificate as valid rather than refuse the connection, because a hard failure would mean every transient network hiccup looked like an attack. An attacker who already controls the network path, which is exactly the position of someone running a man-in-the-middle, can simply block the revocation check and the soft-fail does the rest. So even a CA that knows a certificate is bad cannot reliably stop a browser from accepting it. That is the gap the trust store closes, and the gap that made deleting the entire DigiNotar root the only real option. The mechanics of revocation checking and its fingerprinting side effects are covered in depth in OCSP, OCSP stapling, and the revocation-check fingerprint.

DigiNotar shows what happens when a vendor decides the collateral damage is worth it and pulls the root immediately. Symantec shows what happens when the collateral damage is so large that even Google had to schedule the removal a year out and break it into stages.

DigiNotar, June to September 2011

DigiNotar was a small Dutch CA owned, by 2011, by the US authentication vendor VASCO Data Security. It was not a household name. Its importance came from a single accreditation: it issued PKIoverheid certificates, the certificates the Dutch government used for tax filing, vehicle and real-estate registration, citizen authentication, and a long tail of e-government services. A breach of DigiNotar was therefore a breach of the machinery of the Dutch state.

The intrusion began earlier than anyone realized. The Fox-IT investigation later put the first signs of compromise around 6 June 2011. DigiNotar’s own systems detected an intrusion on 19 July. By then the attacker had already been inside the CA software for weeks. Between 10 and 20 July the attacker used that access to issue rogue certificates, including a wildcard certificate for Google dated 10 July. DigiNotar found and revoked many of the bad certificates in its initial cleanup, but it missed some, and critically it did not tell anyone outside the company.

6 Jun intrusion begins 10 Jul *.google.com cert 19 Jul internal detection 28-29 Aug MITM in Iran, public disclosure 3 Sep govt takeover 20 Sep bankrupt *The breach ran undetected for weeks, the fraudulent Google certificate sat unrevoked, and once it surfaced the company lasted under a month.*

The certificate that surfaced in Iran

The rogue *.google.com certificate would have stayed a footnote if it had never been used. It was used. Starting around 28 August 2011, users on multiple Iranian ISPs began hitting certificate warnings, and one of them posted about it to a Google product forum. The certificate they were being served for Google was valid, signed by a CA in every browser’s trust store, and not theirs. Someone was sitting in the middle of the connection.

Google reported the fraudulent certificate publicly on 29 August. Fox-IT, brought in to investigate, found that DigiNotar’s OCSP responder had logged roughly 300,000 distinct IP addresses asking about the rogue Google certificate, and around 99 percent of them were in Iran. That number is the closest thing to a casualty count the incident produced. Each of those addresses is a browser that connected to what it thought was Gmail, through a man-in-the-middle holding a certificate that browsers accepted without complaint. The traffic pattern, including a slice routed through Tor exit nodes by Iranians evading censorship, points at a state-scale interception operation against a specific population.

The final tally from the Black Tulip report was 531 fraudulent certificates across 344 domains, covering not just Google but Skype, Mozilla’s add-ons site, Microsoft Update, the CIA, Mossad, and others. The forensics on DigiNotar’s own infrastructure were damning. The network was poorly segmented, so a foothold in one place spread. Servers ran outdated software with no server-side antivirus. Passwords were weak enough to brute-force. There was no secure central logging, which is why Fox-IT could never fully establish what the attacker had touched. For a company whose entire product was trustworthy signatures, the signing environment was indefensible.

The detail that turned a serious breach into a fatal one was the silence. DigiNotar detected the intrusion on 19 July, ran an internal cleanup, revoked the bad certificates it could find, and decided the matter was handled internally. It did not notify the browser vendors whose trust stores carried its root, did not notify the Dutch government whose e-government ran on its certificates, and did not assume that the certificates it had failed to find would be used. All three assumptions were wrong. The *.google.com certificate it missed was the one deployed against Iranian users six weeks later. Had DigiNotar disclosed in July, the vendors could have started revoking trust before any interception happened, and the company might have survived as a chastened CA rather than a defunct one. The lesson the industry drew, and later wrote into the Baseline Requirements as mandatory incident reporting, is that a CA’s obligation on discovering a compromise is to tell the people who depend on it, immediately, even at the cost of admitting it failed. Concealment converts a recoverable incident into an unrecoverable one.

Collapse

Once the fraudulent certificate was public, trust evaporated faster than DigiNotar could respond. On 29 August Mozilla and Microsoft moved to remove DigiNotar from their trusted-certificate lists. Mozilla shipped updated Firefox builds (3.6.21, 6.0.1, and the 7, 8, and 9 branches) plus Thunderbird and SeaMonkey to revoke trust in the DigiNotar root. Apple followed with Security Update 2011-005 for Mac OS X on 9 September and patched iOS on 13 October. Google and Microsoft pushed their own removals. Within roughly two weeks of the certificate surfacing, DigiNotar’s roots were being purged from every major platform.

The Dutch government had a harder problem, because DigiNotar’s PKIoverheid certificates ran live e-government. Pulling them instantly would have stopped tax filing and parts of the judicial system cold. The Dutch Safety Board’s later report was blunt that an immediate revocation would have caused major social and economic disruption, so the state took operational control of DigiNotar on 3 September and managed an orderly migration off its certificates that took months. The accreditation to issue qualified certificates was withdrawn. On 19 September DigiNotar filed for bankruptcy, and the Court of Haarlem declared it bankrupt on 20 September 2011. VASCO later put its direct losses from the bankruptcy under five million dollars, a small number for the destruction of a CA, because the asset that died was reputation and reputation does not show up cleanly on a balance sheet.

A person using the handle Comodohacker claimed responsibility, the same handle that had claimed the Comodo reseller breach in March 2011, and Fox-IT noted tool and technique overlap between the two intrusions. Whether that was one actor or a convenient flag is still unsettled. What is settled is the lesson the industry took: a single compromised CA can issue a working certificate for any domain on the internet, and the only fast remedy is to delete the CA.

What DigiNotar changed

The DigiNotar collapse is the event most often cited as the reason Certificate Transparency exists. The problem it exposed is structural. A CA can issue a certificate for any domain, and before 2013 there was no public, tamper-evident record of what had been issued. The fraudulent Google certificate sat unrevoked because nobody outside the attack path knew it existed. Google’s answer, drafted in the aftermath and published as RFC 6962 in June 2013, was to require certificates to be logged to append-only, cryptographically verifiable public logs. If every certificate is logged, then a domain owner can watch the logs and see a mis-issuance the moment it happens, instead of finding out when a browser in Iran throws a warning.

CT did not arrive overnight, and its enforcement story is the bridge to the Symantec case. Chrome began requiring CT for Extended Validation certificates in 2015 and broadened the requirement over the following years until, by April 2018, every newly issued publicly trusted certificate had to be logged or Chrome would reject it. The enforcement matters as much as the logs. A log nobody is required to use catches nothing; a log every CA must write to, backed by a browser that refuses unlogged certificates, turns issuance into a public record that domain owners and researchers can audit continuously. That requirement is exactly what surfaced Symantec’s problems. The mechanism built to catch the next DigiNotar caught the largest CA in the world instead, and caught it not doing something dramatic like a breach, but quietly issuing certificates it should never have issued. The 2015 Thawte test certificate for Google’s own domain was found because it showed up in a CT log that Google was watching. Without that log, it would have been invisible in exactly the way the DigiNotar certificate was invisible until a browser threw a warning in Tehran. For the full history of CT logs and how the gossip and audit model works, see certificate transparency; for the wider arc of CA disasters that drove these reforms, the history of certificate authorities covers the run-up.

DigiNotar Symantec cause timeline collateral outcome key compromise ~2 weeks to purge all certs broke bankruptcy policy mis-issuance ~18 months, staged ~30% of HTTPS web sold to DigiCert *Same endpoint, opposite mechanics: one CA died of a break-in in two weeks, the other of a policy verdict carried out over a year and a half.*

Symantec, 2015 to 2018

Symantec’s CA business was not small. Through the Symantec, Thawte, GeoTrust, RapidSSL, and VeriSign brands it had bought, the company was the largest commercial issuer of public TLS certificates, with estimates at the time putting its share of issued certificates somewhere around a third of the public web. Distrusting it was not like distrusting a niche Dutch CA. It meant breaking a large fraction of HTTPS sites unless every one of them swapped certificates first. That scale is why the Symantec case played out as negotiation rather than emergency.

The trouble started in 2015. Through Certificate Transparency logs, Google noticed that Symantec’s Thawte brand had issued an Extended Validation pre-certificate for google.com and www.google.com that Google had never requested. These were described as test certificates, issued internally and never meant to leave the building, but they were real certificates chaining to a trusted root, and one of them was for someone else’s domain. An initial audit put it at a couple dozen test certificates across domains owned by Google, Opera, and others. A deeper look found more: an additional 164 certificates across 76 domains had been issued the same careless way. Symantec fired several employees involved and published reports on the scope.

That should have been the floor. It turned out to be a sample. The practice of issuing unauthorized test certificates for domains the company did not control had been going on, on and off, from roughly 2009 onward, continuing even after the Baseline Requirements that took effect in July 2012 prohibited exactly that kind of issuance. The 2015 incident was the visible part of a pattern, and the pattern is what eventually cost Symantec the business.

The 2017 escalation

In January 2017 a new round of investigation surfaced a different and arguably worse problem. Symantec did not validate and issue every certificate itself. It delegated authority to registration authorities, third-party partners who could approve issuance, including CrossCert in Korea, plus Certisign, Certsuperior, and Certisur. An audit of CrossCert found roughly 127 mis-issued certificates, with domain names that were never validated and compliance warnings overridden by staff without review. The RA model meant Symantec had handed the power to issue publicly trusted certificates to partners it was not adequately supervising.

The delegation matters because it changes what a trust decision is about. When Symantec validated and signed a certificate itself, a browser trusting Symantec was trusting one organization’s controls. When Symantec let CrossCert approve issuance, the browser was transitively trusting CrossCert’s controls, and Certisign’s, and every other RA’s, often without knowing those parties existed. A trust store inclusion is supposed to be a statement about a known, audited entity. The RA model quietly turned it into a statement about a tree of partners the vendors had never reviewed. That is a structural objection independent of how many certificates were actually mis-issued, and it is the objection that did the most damage to Symantec’s position, because there was no clean way to argue it away.

This is where the dispute turned sharp. On 23 March 2017, on Chrome’s blink-dev mailing list, Google’s Ryan Sleevi posted an “Intent to Deprecate and Remove” trust in Symantec-issued certificates, co-authored with Devon O’Brien and Andrew Whalley of Chrome Security. The post laid out a series of findings that, in Google’s reading, showed Symantec had repeatedly failed to uphold the policies a public CA is bound by, going back years, with the RA program as the latest and largest example. Google’s number for the full scope was at least 30,000 certificates of questionable provenance. The early proposals were aggressive: shrink the maximum accepted validity of Symantec certificates with each Chrome release so that long-lived certificates would be forced to reissue quickly, and strip the green Extended Validation indicator from Symantec EV certificates immediately, since the whole premise of EV is rigorous validation and that premise was what had failed.

Symantec’s response was to dispute both the count and the severity. The company argued the genuinely improper test certificates numbered in the dozens, not the tens of thousands, and that no real-world harm had occurred, since unlike DigiNotar there was no evidence any of these certificates was used to attack anyone. Both things can be true at once. No one was intercepted, and Symantec had still issued certificates outside the rules at a scale and for a duration that meant the browsers could no longer treat its assertions as reliable. The browsers’ position was that a public CA’s value is precisely the assertion that it follows the rules, and a CA that doesn’t is worthless even if no specific certificate has yet been abused.

The staged distrust that shipped

The debate ran for months on blink-dev and on Mozilla’s policy lists, with proposals and counter-proposals about how much to break and how fast. The outcome, reached over the summer of 2017 and laid out in Google’s 11 September 2017 post by O’Brien, Sleevi, and Whalley, was a managed transition rather than an instant purge. Symantec would stop running its own issuance and move to a new, independently operated “Managed Partner Infrastructure.” Symantec chose to sell the CA business outright rather than rebuild it, and in August 2017 agreed to sell it to DigiCert for about 950 million dollars. DigiCert closed the acquisition by the end of October 2017 and began issuing certificates from the new infrastructure on 1 December 2017. Certificates issued from that date forward chained to DigiCert and were never in question.

The old Symantec certificates were distrusted in two Chrome stages, keyed to the date each certificate was issued.

Chrome distrust, by certificate issuance date issued before 1 Jun 2016 distrusted in Chrome 66 stable 17 Apr 2018 issued 1 Jun 2016 - 1 Dec 2017 distrusted in Chrome 70 stable 23 Oct 2018 issued from 1 Dec 2017 DigiCert root, fully trusted new Managed Partner Infrastructure *Chrome 70's stable release on 23 October 2018 removed trust in the old Symantec-rooted infrastructure entirely; anything still chaining to it broke on update.*

Chrome 66, which reached stable on 17 April 2018 after a 15 March beta, removed trust in Symantec certificates with a not-before date earlier than 1 June 2016. Chrome 70, stable on 23 October 2018 after a 13 September beta, removed trust in the remaining old Symantec-rooted infrastructure. Mozilla ran a parallel schedule in Firefox, with console warnings in Firefox 58, hard untrusted-connection errors in Firefox 60 around May 2018, and full root removal in Firefox 63 that October. Site operators had a clear instruction: replace any Symantec-brand certificate with one from the new DigiCert infrastructure, or from any other CA, before the relevant deadline.

The migration was large but orderly precisely because it was scheduled. Operators had over a year of notice, a hard cutoff, and a drop-in replacement from the same vendor under new management. The contrast with DigiNotar is the whole point. DigiNotar’s certificates broke in two weeks because the CA could not be trusted for even one more day. Symantec’s broke on a published calendar because the failure was a policy and governance verdict, not a live compromise, and the cost of breaking a third of the web instantly was unthinkable when a staged removal would do.

What it actually takes to revoke a CA

Strip both stories down and the same machine is running underneath. A browser vendor maintains a root store. When a CA can no longer be trusted, the vendor removes the root, or constrains it, and ships the change in a software update. Everything downstream of that root stops validating on the next update the user installs. There is no negotiation with the CA at the moment of removal, no court order required, no protocol handshake. The vendor decides, the update ships, the trust is gone. This is the actual backstop of the web PKI, and both DigiNotar and Symantec are demonstrations of it firing.

What differs is the trigger and the tempo. DigiNotar pulled the trigger on a confirmed key compromise with active exploitation in the wild, and the vendors acted in days because every additional day was additional interception. The collateral damage, every legitimate DigiNotar certificate breaking at once, was accepted because the alternative was worse, and the one place it could not be accepted (Dutch e-government) got a hand-managed migration instead of an instant cut. Symantec pulled the trigger on a long record of mis-issuance and weak oversight with no proven exploitation, and the vendors acted over eighteen months because the collateral damage of an instant cut, roughly a third of HTTPS, was itself the larger risk. One is the emergency mode of trust revocation, the other is the deliberate mode. The system has both because failures come in both shapes.

The harder lesson is about asymmetry. A CA spends years and real money earning inclusion in the trust stores, passing audits, building infrastructure, signing customers. It can lose all of it in two weeks, or be sentenced to lose it on a one-year clock, by a decision made on a public mailing list. Trust here is expensive to acquire and cheap to destroy, and that is by design, because the alternative, a CA too big or too entangled to distrust, is the failure mode that ends the system entirely. Symantec was as close to too-big-to-distrust as a CA has come, a third of the web riding on its roots, and it was distrusted anyway. That precedent, more than any single revoked certificate, is what keeps the other CAs honest. The padlock means something only because the browsers have shown, twice, that they will take it away.

Sources & further reading

Further reading