March 29 Microsoft developer Andres Freund was trying to optimize his computer’s performance when he noticed that a program was using an unexpected amount of processing power. Freund got involved in solving the problem and “suspected”.
Eventually, Freund found the source of the problem, which he then posted on a security mailing list: he discovered a backdoor in XZ Utils, a data compression utility used by a wide variety of Linux-based computer applications—a constellation of open-source software that, while often not aimed at consumers, it supports key computing and Internet functions such as secure machine-to-machine communication.
By chance spotting the backdoor, which was buried deep in the code in the binary test files, Freund avoided a large-scale security disaster. Any machine with an operating system that included the backdoored utility and met the specifications specified in the malicious code would be vulnerable to compromise, allowing an attacker to potentially take control of the system.
The XZ backdoor was introduced through what is known as a software supply chain attack, which the National Counterintelligence and Security Center defines as “deliberate actions directed against the supply chains of software products themselves.” Attacks often use complex means of altering a program’s source code, such as gaining unauthorized access to a developer’s system or through a malicious insider with legitimate access.
The malicious code in XZ Utils was introduced by a user named Jia Tan, using the handle JiaT75, according to Ars Technica and Wired. Tan has been involved with the XZ project since at least late 2021 and has built the trust of the developer community working on it. Eventually, though the exact timeline is unclear, Tan became a co-maintainer of the project, alongside the founder, Lasse Collin, allowing Tan to add code without needing to approve contributions. (Neither Tan nor Collin responded to requests for comment.)
The back door of the XZ betrays sophisticated, meticulous work. First, whoever led the attack identified a piece of software that would be embedded in a wide variety of Linux operating systems. Development of this widely used technical utility was understaffed, with one lead maintainer, Collin, later admitting he was unable to maintain XZ, providing an opportunity for another developer to step in. Then, after gaining Collin’s trust over a period of years, Tan slipped a backdoor into the utility. All of these moves were subject to technical expertise that led to the creation and embedding of actual backdoor code — code sophisticated enough that analysis of its precise functionality and capabilities is still ongoing.
“The care taken to hide exploits in test binaries, and just the time it takes to build a reputation in an open-source project to exploit later, is abnormally sophisticated,” said Molly, a systems administrator at the Electronic Frontier Foundation who goes a mononym. “However, there is no indication yet whether this was sponsored by a state, a hacker group, a rogue developer, or any combination of the above.”
Tan’s rise to the position of co-maintainer mostly took place in an email group where code developers — in the open-source collaborative spirit of the Linux family of operating systems — exchange ideas and devise strategies for building applications.
On one email list, Collin faced a barrage of complaints. A group of users, relatively new to the project, protested that Collin was lagging behind and not updating the software fast enough. He should, some of those users said, hand over control of the project; some specifically requested the addition of another maintainer. Admitting that he could no longer devote sufficient attention to the project, Collin made Tan a co-maintainer.
The users involved in the complaints seemed to materialize out of nowhere — posting their messages from what appeared to be recently created Proton Mail accounts and then disappearing. Their entire online presence is tied to these brief interactions on the XZ dedicated mailing list; their only recorded interest is in rapidly rolling out software updates.
Various US intelligence agencies have recently expressed interest in addressing software supply chain attacks. The Cybersecurity and Infrastructure Security Agency jumped into action after Freund’s discovery, issuing a warning about the XZ backdoor on March 29, the same day Freund went public about it.
Open source players
In the open source world of Linux programming—and in XZ Utils development—collaboration takes place through email groups and code repositories. Tan posted on the listserv, spoke with Collin, and contributed code changes to the code repository Github, which is owned by Microsoft. GitHub has since disabled access to the XZ repository and disabled Tan’s account. (In February, The Intercept and other digital news companies sued Microsoft and its partner OpenAI for using their journalism without permission or credit.)
Several other people on the email list participated in the effort—which seemed diffuse but coincided in its goals and timing—to install a new co-maintainer, sometimes specifically advocating for Tan.
Later, on a listserv dedicated to Debian, one of the more popular operating systems from the Linux family, another group of users advocated the inclusion of a backdoored version of XZ Utils in the distribution of the operating system.
These dedicated groups played different roles: in one case, they complained about the lack of progress on XZ Utils and demanded faster updates by installing a new co-maintainer; and, in the second case, advocating for the rapid and widespread distribution of updated versions.
“I think multiple green accounts that appear to be coordinated around specific goals at key moments fit the pattern of using sock account networks for social engineering that we’ve seen across social media,” said Molly, the EFF’s system administrator. “It is very possible that a rogue developer, hacking group, or government sponsor used this tactic as part of their backdoor plan. Of course, it’s also possible that these are just coincidences.”
The pattern appears to correspond to what is known in intelligence parlance as “identity management,” the practice of creating and subsequently maintaining multiple fictitious identities. A leaked document from defense contractor HBGary Federal outlines the meticulousness that might go into maintaining these fictitious personas, including creating an elaborate online footprint — something the accounts included in the XZ timeline definitely lacked.
While these other users used different email addresses, in some cases they used providers that provide clues as to when their accounts were created. For example, when they used Proton Mail accounts, the encryption keys associated with those accounts were generated on the same day or a few days before the user’s first posting to the email group. (However, users can also generate new keys, which means that email addresses may have been older than their current keys.)
One of the earliest users on the list used the name Jigar Kumar. Kumar appears on the XZ development mailing list in April 2022, complaining that some of the tool’s features are confusing. Tan promptly responded to the comment. (Kumar did not respond to a request for comment.)
Kumar has repeatedly surfaced with subsequent complaints, sometimes building on others’ displeasure. After Dennis Ens appeared on the same mailing list, Ens also complained about the lack of response to one of his messages. Collin admitted that things were piling up and mentioned that Tan was helping him off the list; he may soon have “a bigger role in XZ Utils.” (Ens did not respond to a request for comment.)
After another complaint from Kumar that he was looking for a new carer, Collin replied: “I haven’t lost interest but my ability to care has been quite limited mainly due to long term mental health issues but also some other things. I’ve been doing some off-the-charts work with Jia Tan on XZ Utils recently and maybe he’ll have a bigger role in the future, we’ll see.”
The pressure kept coming. “As I hinted in earlier emails, Jia Tan may have a bigger role in the project in the future,” Collin replied after Ens suggested he hand over some responsibilities. “He helped a lot outside the list and is practically already a co-maintainer. :-)”
Ens then went silent for two years — resurfacing around the time the bulk of the malicious backdoor code was installed in the XZ software. Ens continued to call for faster updates.
After Collin finally made Tan a co-maintainer, there was added pressure to get XZ Utils — which until now had a backdoor — widely distributed. After first appearing on the XZ GitHub repository in June 2023, another figure calling himself Hans Jansen followed up this March to push for a new version of XZ to be included in Debian Linux. (Jansen did not respond to a request for comment.)
An employee of Red Hat, an IBM-owned software company that sponsors and helps maintain Fedora, another popular Linux operating system, described how Tan tried to convince him to help add the compromised XZ Utils to Fedora.
These popular Linux operating systems account for millions of computer users — meaning a huge number of users would be open to compromise if Freund, the developer, hadn’t discovered the backdoor.
“While the possibility of social engineering of backdoors in critical software seems like an indictment of open source projects, it is not exclusive to open source and can happen anywhere,” Molly said. “In fact, the engineer’s ability to discover this backdoor before it was shipped was only possible because of the open nature of the project.”