How Was the XZ Utils Backdoor Discovered?

TL;DR

A Microsoft developer uncovered a hidden backdoor in XZ Utils after noticing that SSH consumed unusual CPU resources and started about 500 milliseconds slower. His report reached the Debian security team before the compromised software spread widely, allowing developers and cybersecurity experts to neutralize code that could have enabled unauthorized remote access to countless Linux systems.

Transcript

Andress is running some routine software tests at first everything looks pretty normal but then he spots something this peak here is not a typical CPU load not a huge problem but it's annoying plus he can't make sense of it right away little does andrees know that this is not just some random bug it's a cleverly hidden back door one that's been pla... Read More

Key Insights

• XZ Utils is a data compression tool included in nearly all major Linux distributions, where it helps package data for faster transfer and storage. Its broad presence made a malicious modification especially dangerous because Linux powers most servers, mainframes, and every top 500 supercomputer described in the transcript.
• Open-source maintainers are responsible for reviewing proposed changes and deciding what enters a project. Lasse Collin had performed this role for XZ Utils since developing the tool in 2005, but the unpaid workload and his limited capacity created an opportunity for another contributor to gain influence.
• Jia Tan gained trust through regular, polite contributions beginning in 2022 after appearing on GitHub in January 2021 and contributing to other projects for over a year. Tan became co-maintainer in June 2022, then obtained increasing authority to edit XZ Utils independently.
• Security reporting for XZ Utils changed in March 2023 when Tan redirected the project's OSS-Fuzz contact address. After that change, security alerts from the service, which identifies bugs in open-source projects, went exclusively to Tan instead of reaching Collin.
• The compromised update reached Debian Sid after Hans Jansen requested its inclusion and other accounts praised the supposed bug fix or dismissed objections. Sid is a preliminary Debian version labeled unstable and insecure, giving developers an opportunity to test changes before their official integration.
• Andres Freund discovered the problem because SSH used unusually high CPU resources and started approximately 500 milliseconds slower during his tests. He traced the performance regression through SSH's dependency on XZ Utils and found malicious code concealed within two versions of the compression software.
• The hidden code created a backdoor that could allow an attacker to enter Linux systems through SSH. Its disguise helped it evade code review and testing, but that same concealment reduced performance enough for Freund to notice and investigate the otherwise subtle delay.
• The response began after Freund contacted the Debian security team on March 27 and published his findings several days later. Developers and cybersecurity experts produced a fix within hours, while the compromised software remained largely confined to test versions of several major Linux distributions.

Questions & Answers

Q: How was the XZ Utils backdoor discovered?

Microsoft developer Andres Freund discovered the backdoor while testing software on Debian Sid. He noticed that SSH consumed an unusual amount of CPU resources and started approximately 500 milliseconds slower than expected. Because SSH depended on XZ Utils on the affected Linux distribution, he investigated that connection and uncovered malicious code hidden inside two versions of XZ Utils.

Q: What could the XZ Utils backdoor have allowed attackers to do?

The malicious code could have enabled unauthorized remote access through SSH on affected Linux systems. If compromised XZ Utils versions had entered multiple operating systems and spread broadly, attackers could have accessed countless computers and servers, disrupted critical services, distributed malware, or stolen sensitive information from organizations such as hospitals, schools, universities, companies, and governments.

Q: Why was XZ Utils an important target for attackers?

XZ Utils was a valuable target because it is a data compression tool included in nearly every major Linux distribution described in the transcript. Linux powers most of the world's servers and mainframes, along with every one of the top 500 supercomputers. Compromising a trusted component with such broad distribution could therefore expose an enormous number of connected systems.

Q: How did Jia Tan gain control over the XZ Utils project?

Jia Tan first appeared on GitHub in January 2021, contributed to other projects for more than a year, and began regularly submitting helpful, polite changes to XZ Utils in 2022. Tan became co-maintainer in June 2022, gradually edited the project independently, and changed its OSS-Fuzz contact address in March 2023 so security alerts went exclusively to Tan.

Q: How did social engineering contribute to the XZ Utils attack?

The attack involved accounts that appeared to pressure maintainer Lasse Collin, advocate for Jia Tan, and promote the compromised release. Dennis Ens and Jigar Kumar criticized Collin's slow responses and suggested that the project needed another maintainer. Later, Hans Jansen requested the release's inclusion in Debian, while other users praised the fix or dismissed concerns about the upload.

Q: Why did the malicious XZ Utils release reach Debian Sid?

Hans Jansen asked the Debian project to include a new XZ Utils version, claiming that it fixed a specific bug. Two other users supported the proposed fix, and another account quickly dismissed a concern raised during the discussion. Debian then integrated the version into Sid, its preliminary distribution marked unstable and insecure for testing before wider release.

Q: What did Andres Freund do after finding the backdoor?

After tracing the SSH performance problem to malicious code in XZ Utils, Andres Freund emailed the Debian security team on March 27. A few days later, he disclosed his findings publicly in a Mastodon post. The disclosure prompted an emergency security alert and a rapid response from developers and cybersecurity experts, who created a fix within hours.

Q: Why did the XZ Utils backdoor fail to spread widely?

The backdoor was detected while the compromised software was still present mainly in test versions of several major Linux distributions. Its concealed implementation introduced abnormal CPU usage and an approximately 500-millisecond SSH delay, which caught Andres Freund's attention. His investigation and disclosure gave developers time to neutralize the malicious code before the affected versions entered broad deployment.

Summary & Key Takeaways

• XZ Utils is a widely included Linux compression tool originally developed and maintained by Lasse Collin. Because maintaining it was an unpaid hobby and Collin described limited capacity related partly to long-term mental health issues, contributor Jia Tan gradually gained trust, became co-maintainer in June 2022, and assumed increasing control over the project.

• The attack combined malicious code with sustained social engineering. Several accounts pressured Collin, supported Tan, promoted a compromised XZ Utils release, and dismissed concerns. After Debian placed the release in its unstable Sid distribution, Microsoft developer Andres Freund encountered it while testing another open-source project and investigated abnormal SSH performance and CPU consumption.

• Freund traced an approximately 500-millisecond SSH delay to XZ Utils and discovered a concealed backdoor designed to permit unauthorized system access. He alerted Debian security personnel on March 27 and disclosed the problem publicly days later. Experts quickly produced a fix before the affected versions reached broad deployment across major Linux distributions.