I generally don't like writing about current events, because the inherent ephemeral nature of the news means that your words become less meaningful with each passing day. As a professional, I find wasting other people's time in poor taste. However, the topic of supply chain attacks has come up repeatedly in the recent months, so I'd like to take a moment to reflect on these incidents, as well as the work I've done through Paragon Initiative Enterprises (PIE for short) to be able to solve this problem at scale. ## What's a Supply Chain Attack? In general, a supply chain attack involves first hacking a trusted third party who provides a product or service to your target, and then using your newly acquired, privileged position to compromise your intended target. More narrowly, we're only interested in supply chain attacks which involve compromising the companies that produce software used by other companies. Hardware based supply chain attacks have been demonstrated by leaked documents concerning nation state actors. ## A Recent History of Supply Chain Attacks Rather than rehash the long history of supply-chain attacks (both theoretical and actual) and the motivation for virtually every Linux distro to PGP-sign their update information, let's just look at what's happened in the year 2017. The industry's first wake-up call in recent months was a ransomware called NotPetya that was spread throughout Ukraine through [the auto-update mechanism for *M.E.Doc*, one of the few government-approved accounting softwares](https://www.cyberscoop.com/petya-ransomware-medoc-hacked-auto-update/). Last month, Kaspersky Lab's research teams published a write-up about a [malware campaign called ShadowPad](https://www.kaspersky.com/about/press-releases/2017_shadowpad-how-attackers-hide-backdoor-in-software-used-by-hundreds-of-large-companies-around-the-world), which infected a software update for an unspecified product by a company called [NetSarang](https://www.netsarang.com/). The product in question is reportedly used by hundreds of large firms, especially in the energy sector, so the long-term impact of such an infection (if undetected) is best left to the imagination. Most recently, but certainly not finally, we learned that the most recent updates for a common Windows utility called [CCleaner had also been infected with malware](https://techcrunch.com/2017/09/21/ccleaner-supply-chain-malware-targeted-tech-giants/). ### The Common Denominator in Recent Supply-Chain Attacks In each of these recent cases, malicious actors had compromised a software company and infected the "legitimate" copy of the software that users would download and install. In some of these cases, an automatic update mechanism was used to deliver the payloads. However, it is erroneous to call out auto-update features as the reason for infection. If a malicious and signed binary was uploaded to a company website, the fact that updates have to be performed manually would not have helped the victims. They would just happily install it, none the wiser. In all of these cases, the malicious software update was spread to everyone who installed it, rather than a targeted attack against only networks of interest. I'd wager that there are some of these going on in the wild as I type this, but they're inherently much more difficult to detect. The real security problem here has very little to do with automation. ## The Fundamental Problem with Software Update Security The reason that supply chain attacks are viable, effective, and often difficult to detect is that *very little* (if any) industry attention is given towards guaranteeing [Software Authenticity](https://defuse.ca/triangle-of-secure-code-delivery.htm). This is a topic we have covered extensively: * [Quick and dirty guide to secure automatic updates](https://paragonie.com/blog/2017/08/quick-guide-simple-and-secure-automatic-updates) (August 2017) * [A technical tutorial on the ins and outs of automatic update implementations for PHP developers](https://paragonie.com/blog/2016/10/guide-automatic-security-updates-for-php-developers) (October 2016) * [A detailed write-up of the implementation in our Free Software content management system, CMS Airship](https://paragonie.com/blog/2016/05/keyggdrasil-continuum-cryptography-powering-cms-airship) (May 2016) * [Our very first blog post introduced *Asgard*, our first attempt to solve this problem for the PHP community](https://paragonie.com/blog/2015/04/introducing-asgard-authentic-software-guard) (April 2015) If you read only one of the above links, read the one from August 2017. Our company has done a lot of work between client engagements to make it easier for developers to make their software updates **signed**, **auditable**, and **reproducible**. This does several things: 1. **Signed:** If the signing key is kept offline, even if the webserver or update server is compromised, the malware will not be accepted by the end user's computer. This is a nontrivial impediment to supply chain attacks, provided the company in question has adequate discipline to keep their signing key secret. 2. **Auditable:** Even with a pilfered signing key, with this guarantee in place, attacks will be unavoidably detectable due to each update being committed to an append-only data structure (e.g. a Merkle tree or hash chain). This also ensures that everyone gets the same update and prevents targeted attacks (which, as mentioned above, haven't yet been detected, but cannot be ruled out). 3. **Reproducible:** With open source software and reproducible builds, users can verify that the binary they received corresponds to the expected version of the software they intended to install. Combined with the auditability guarantee, if even one power user or security expert inspects the source code for backdoors, that's sufficient to ensure that all other copies of the same update are authentic. If all three guarantees are provided, it's even [generally safer to deploy your updates automatically](https://paragonie.com/blog/2016/10/guide-automatic-security-updates-for-php-developers#why-automatic-updates) (by default; power users should be allowed to disable this mechanism). Furthermore, because of these guarantees, most attackers will be dis-incentivized to even attempt such an attack: The success of their campaign hinges on their ability to steal a cryptography key, they can't target networks or persons of interest, the likelihood of being caught very early is extraordinarily high. If we can make all software updates come with authenticity guarantees, I believe we will all-but-eliminate supply chain attacks in the real world. ## Does This Really Matter? We've consistently focused on the problem of ensuring software updates are secure enough to automate since our company's inception. We strongly believe that solving this problem at scale will have a more profound impact on the security of the Internet than *almost any other problem in all of information security and applied cryptography*. ### "Wait, I Think I Can Solve This Using [Buzzword]" If you're looking for a killer app cybersecurity solution to these sort of attacks, the answer involves a lot of inglorious janitorial work [cleaning up the software ecosystem](https://meta.stackoverflow.com/questions/293930/problematic-php-cryptography-advice-in-popular-questions) and [assisting open source software projects](https://github.com/composer/packagist/issues/797) to get on board with scalable and trustworthy solutions. This is [not a problem that necessarily needs a blockchain](https://paragonie.com/blog/2017/07/chronicle-will-make-you-question-need-for-blockchain-technology) to solve. It's not a problem that traditional endpoint security (read: anti-virus software) is even in the correct *genre* to a solution. No, we're going to need to solve this as a community, not an industry. That means a lot of dedicated security experts and developers working together. That means a lot of short-term developer hours expended to improve the long-term health of your companies. I fully anticipate that [charlatans and snake-oil salesmen](http://attrition.org/errata/charlatan/) will come out of the woodwork to pitch their poorly thought-out solutions to supply chain attacks as increasing number of computer criminals find them to be an effective way to infect victims. Even more hours will likely be expended on debunking their fraudulent claims and alerting the uninformed of their deception. But in the end? If we, as a community—as a society—can work together to solve the problem of secure software updates for the Internet at large? It will all be worth it.