Bloomberg published an eye-catching story on October 4th entitled “The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies” in which they described how a major US supplier of server hardware has been compromised via the supply chain. According to Bloomberg, a Chinese manufacturer added an additional tiny chip to the server motherboards that they shipped to Supermicro. This additional chip – claimed to be the size of a grain of rice – was supposedly designed and then inserted by the Chinese military as a hardware-based Trojan. The article claims that “When a server was installed and switched on, the microchip altered the operating system’s core so it could accept modifications. The chip could also contact computers controlled by the attackers in search of further instructions and code.” The manipulated mainboards were supposedly used via a US supplier by at least 30 US tech companies.
This story is a “blockbuster” story for everybody concerned with IT security. That is, if the story is true. However, in my opinion, there is still some reason to doubt these claims.
When assessing how likely Bloomberg’s story is, one can look at the article from four different angles:
- Technical: How realistic is it that an agency can find a way to alter the hardware in a stealthy way to include a backdoor at the hardware level?
- Operational: How realistic is it that the Chinese government can place such a Trojan high in the supply chain in a way that the Trojan hardware reaches its intended target?
- Credibility: How credible are the sources and the way the story is published?
- Are there alternative explanations for the suspicious chips and how likely are they?
Technical perspective
From a technical perspective, I think stealthy hardware modifications that enable attackers’ easier access are possible. My personal experience with looking into hardware Trojans from an academic perspective is that there is also some amount of luck involved. Sometimes a solution is easy to find, sometimes it is more difficult. The article is very sparse on technical details, so it is not clear what exactly the inserted chip is supposed to do. So how difficult this was to pull off is hard to judge from the outside and the sparse information in the report. But China has enough expertise and manpower to address such problems that I am sure they will eventually find a solution to compromise a system via a hardware manipulation in a stealthy way.
Operational perspective
The manipulated servers are quite specialized. It is not something you will likely find in someone’s home. With some additional intelligence (either through spies or “traditional” cyber attacks) a secret agency or the manufacturer can make a well-educated guess where the manipulated mainboards are likely to be deployed. It might be hard to manipulate only a few and expect them to reach a single predefined target. But in the end, developing such a Trojan is not the most cost-effective way to infiltrate a single target. Hence, the attacker might not have known for sure where all the manipulated hardware would be used, but he could be certain that it would end up at the intended target or at least in other very valuable places (for China, it seems most US tech companies would be considered valuable targets). So from an operational perspective, this insertion point seems very reasonable to have a good chance that the Trojan ends up where you want it to be.
Credibility
The way this attack is being disclosed via a Bloomberg reporter is more than just a bit suspicious. According to the report, the suspicious chip was discovered in 2015 and resulted in a (secret) investigation. Let us assume that it took some time to analyze the technical details of the attack and confirm it. But by 2016 the US agencies should have been very certain if the chip is a Trojan or not. As a counter-espionage campaign, they might have waited to reveal their discovery to collect evidence or mislead the attackers. However, the report claims that this malicious chip lets China compromise the server’s operating system and hence control the server. And such Trojan infected servers have been deployed in at least 30 US tech companies.
If the chip is not removed or the hardware is exchanged all of these servers at the heart of Silicon Valley and elsewhere in the US are in danger of being controlled by China. Yet, the agency decided to keep quiet and not let these companies know? The article makes it look as if hardware Trojans are the ultimate hacking tool and extremely stealthy. But once you have discovered a hardware Trojan, it is more like the least-stealthy attack. The evidence is there in the hardware for everyone with good eyesight or a magnifying glass to see. Unlike software Trojans that can delete themselves after their job is done, hardware Trojans stay where they have been planted forever. Attribution of the attack is also much easier because developing and manufacturing a Trojan chip is not something a small group can do in a basement.
If the story is true, the US would have the undeniable proof that China manipulates hardware, something the US has been warning about for quite some time. And it would be perfect timing considering the ongoing trade war. By publicly disclosing the chip and details everyone could verify the claim. Academia would *love* to get the hands on those chips to publish tons of papers about what “nation-state hardware Trojans” look like and how one can defend against them. So why not reveal the details? You would have an army of security researchers looking into solutions to detect and prevent such attacks, each one giving scientific credibility to the claims. Why not let every CISO in the US know how to look for such a Trojan chip in their servers? Give advisories how to check the network logs to make sure you have not been a victim in the past? Let the companies help in assessing the real scope of the operation?
Instead, two years later, a handful of secret agents went to talk to a Bloomberg reporter who himself mentioned that he did not know much about hardware Trojans before investigating the story. This seems to make no sense if the malicious chips do what is claimed.
Alternative explanation?
I am not saying that the first part of the story is necessarily wrong: A third party finds a suspicious additional chip during a security audit which prompted an FBI investigation. The point is that there are multiple legitimate reasons why a piece of hardware might have additional or different chips. Take a look at this good blog post that makes the same point. For example, there might have been quality assurance issues during manufacturing or with components used during assembly. So the manufacturer decided, as a quick and cheap fix, to add another component to increase reliability. This fix might have been temporary so that only a limited number of boards were affected. As the blog post noted, it might have been due to issues of the quality of used ICs. Or maybe even a shortage of a specific IC so that they needed to use this workaround to be able to meet the delivery deadlines.
So what is more realistic: That in 2015 the US agencies discovered a large-scale Chinese espionage operation based on additional malicious chips in servers used by US companies and then stayed quiet until 2018 to tip off a reporter without providing technical details or evidence? And during this time the US companies are exposed to the attack? Actually, they still are, since the provided technical details are not enough for administrators to make sure that their servers are not affected by the attack.
Or that in 2015 a security company found a suspicious chip that led to an investigation. An investigation that revealed it was a harmless chip. But instead of telling everyone involved that it was harmless, the US agencies used it to blame China to be a villain in cyber security – right in the middle of a trade war and escalating rhetoric.
I personally believe the latter – which does not mean that it is unrealistic that China (and the US!) manipulates hardware. But if they do, they likely did not do it in this specific case.
PS:
For Forbes, Anna-Katrina Shedletsky also explains that changes by the manufacturer without necessarily the customer’s knowledge is nothing too uncommon: “Even if the exploit isn’t in the official drawing provided by the manufacturer, parts could be made according to a similar, but different, internal-only design. This leads to a reality where you have a “customer-facing drawing” and a “factory drawing”.”
Addition:
Bloomberg ran a new story about hardware manipulations of Super Micro server boards by China. The short version: this story is even less likely.
It is solely based on a story told by a CEO of a company (Sepio) that sells services to detect hardware manipulations. This company would directly gain a lot if i) people were scared of hardware manipulations and ii) people believed that it (i.e. Sepio) can detect such manipulations.
From the Bloomberg article:
“ They [Sepio’s team] decided it represented a serious security breach, along with multiple rogue electronics also detected on the network, and alerted the client’s security team in August, which then removed them for analysis. Once the implant was identified and the server removed, Sepio’s team was not able to perform further analysis on the chip.“
In other words, Sepio has a system that flags hardware as suspicious. In this case, their system flagged Super Micro hardware as suspicious. The result was that the server was replaced not by Sepio but the company’s security team. However, “Sepio’s team was not able to perform further analysis on the chip.”
So how can Appleboum be certain that it was a malicious hardware implant and not what we call a “false positive” in scientific discussion of hardware Trojan detection techniques? A “false positive” is when a detection system flags something legitimate as malicious hardware. Appleboum acknowledges himself that the chip/hardware itself was not analyzed to rule out a false positive. I would be very surprised if Sepio created a hardware implant detection system that can identify malicious components on live-running systems without any false positives.
To conclude, I totally believe that Sepio’s system flagged a Super Micro server at a telecommunication company as being a stealthy Chinese hardware implant. But to me, it is much more likely that it was a false positive than that it was a Chines hardware implant. (How could they know it was a Chinese implant if they did not even know how exactly the hardware was manipulated?).
If one needs to go to Bloomberg business news with such a story and no proofs… well, no publicity is bad publicity I guess … even in an industry that relies on trust.