XZ Utils Backdoor

[Zema Bus]

Researchers have found a malicious backdoor in a compression tool that made its way into widely used Linux distributions, including those from Red Hat and Debian.

The compression utility, known as xz Utils, introduced the malicious code in versions ​​5.6.0 and 5.6.1, according to Andres Freund, the developer who discovered it. There are no known reports of those versions being incorporated into any production releases for major Linux distributions, but both Red Hat and Debian reported that recently published beta releases used at least one of the backdoored versions—specifically, in Fedora Rawhide and Debian testing, unstable and experimental distributions. A stable release of Arch Linux is also affected. That distribution, however, isn't used in production systems.

Because the backdoor was discovered before the malicious versions of xz Utils were added to production versions of Linux, “it's not really affecting anyone in the real world,” Will Dormann, a senior vulnerability analyst at security firm Analygence, said in an online interview. “BUT that's only because it was discovered early due to bad actor sloppiness. Had it not been discovered, it would have been catastrophic to the world.”

Several people, including two Ars readers, reported that the multiple apps included in the HomeBrew package manager for macOS rely on the backdoored 5.6.1 version of xz Utils. HomeBrew has now rolled back the utility to version 5.4.6. Maintainers have more details available here.
Breaking SSH authentication

The first signs of the backdoor were introduced in a February 23 update that added obfuscated code, officials from Red Hat said in an email. An update the following day included a malicious install script that injected itself into functions used by sshd, the binary file that makes SSH work. The malicious code has resided only in the archived releases—known as tarballs—which are released upstream. So-called GIT code available in repositories aren’t affected, although they do contain second-stage artifacts allowing the injection during the build time. In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.

The malicious changes were submitted by JiaT75, one of the two main xz Utils developers with years of contributions to the project.

“Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system,” Freund wrote. “Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the ‘fixes’” provided in recent updates. Those updates and fixes can be found here, here, here, and here.

On Thursday, someone using the developer's name took to a developer site for Ubuntu to ask that the backdoored version 5.6.1 be incorporated into production versions because it fixed bugs that caused a tool known as Valgrind to malfunction.

“This could break build scripts and test pipelines that expect specific output from Valgrind in order to pass,” the person warned, from an account that was created the same day.

One of maintainers for Fedora said Friday that the same developer approached them in recent weeks to ask that Fedora 40, a beta release, incorporate one of the backdoored utility versions.

“We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added),” the Ubuntu maintainer said. "He has been part of the xz project for two years, adding all sorts of binary test files, and with this level of sophistication, we would be suspicious of even older versions of xz until proven otherwise."

Maintainers for xz Utils didn’t immediately respond to emails asking questions.

The malicious versions, researchers said, intentionally interfere with authentication performed by SSH, a commonly used protocol for connecting remotely to systems. SSH provides robust encryption to ensure that only authorized parties connect to a remote system. The backdoor is designed to allow a malicious actor to break the authentication and, from there, gain unauthorized access to the entire system. The backdoor works by injecting code during a key phase of the login process.

“I have not yet analyzed precisely what is being checked for in the injected code, to allow unauthorized access,” Freund wrote. “Since this is running in a pre-authentication context, it seems likely to allow some form of access or other form of remote code execution.”

In some cases, the backdoor has been unable to work as intended. The build environment on Fedora 40, for example, contains incompatibilities that prevent the injection from correctly occurring. Fedora 40 has now reverted to the 5.4.x versions of xz Utils.

Xz Utils is available for most if not all Linux distributions, but not all of them include it by default. Anyone using Linux should check with their distributor immediately to determine if their system is affected. Freund provided a script for detecting if an SSH system is vulnerable.

From arstechnica.com

[Grogan]

That's pretty bad when you can't trust a project's developer. Maybe he was compromised.

Arch has this fixed with 5.6.1-2 pkgrels.

P.S. In looking at the PKGBUILD, they switched to an upstream git repo, because the compromised github one is locked. They also removed the malcious dev's keys so they can't be used to verify code.

[Zema Bus]

I read that they're now reviewing the other commits that developer made in other projects.

[Zema Bus]

The backstory for this backdoor finally came out. I've been hearing suspicion that the developer who took control of the project is working on the behalf of a government.

The story of the XZ backdoor’s discovery starts in the early morning of March 29th, as San Francisco-based Microsoft developer Andres Freund posted on Mastodon and sent an email to OpenWall’s security mailing list with the heading: “backdoor in upstream xz/liblzma leading to ssh server compromise.”

Freund, who volunteers as a “maintainer” for PostgreSQL, a Linux-based database, noticed a few strange things over the past few weeks while running tests. Encrypted log-ins to liblzma, part of the XZ compression library, were using up a ton of CPU. None of the performance tools he used revealed anything, Freund wrote on Mastodon. This immediately made him suspicious, and he remembered an “odd complaint” from a Postgres user a couple of weeks earlier about Valgrind, Linux’s program that checks for memory errors.

After some sleuthing, Freund eventually discovered what was wrong. “The upstream xz repository and the xz tarballs have been backdoored,” noted Freund in his email. The malicious code was in versions ​​5.6.0 and 5.6.1 of the xz tools and libraries.

Shortly after, enterprise opensource software company Red Hat sent out an emergency security alert for users of Fedora Rawhide and Fedora Linux 40. Ultimately, the company concluded that the beta version of Fedora Linux 40 contained two affected versions of the xz libraries. Fedora Rawhide versions likely received versions 5.6.0 or 5.6.1 as well.

Freund later identified the person who submitted the malicious code as one of two main xz Utils developers, known as JiaT75, or Jia Tan. “Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the “fixes” mentioned above,” wrote Freund in his analysis, after linking several workarounds that were made by JiaT75.

JiaT75 was a familiar name: they’d worked side-by-side with the original developer of .xz file format, Lasse Collin, for a while. As programmer Russ Cox noted in his timeline, JiaT75 started by sending apparently legitimate patches to the XZ mailing list in October of 2021.

Other arms of the scheme unfolded a few months later, as two other identities, Jigar Kumar and Dennis Ens, began emailing complaints to Collin about bugs and the project’s slow development. However, as noted in reports by Evan Boehs and others, “Kumar” and “Ens” were never seen outside the XZ community, leading investigators to believe both are fakes that existed only to help Jia Tan get into position to deliver the backdoored code.

From theverge.com

[Grogan]

Yes, with a name like Jia Tan he has to be a Chinese spy

(No, I mean he was playing the long game, a simple miscreant hacker isn't going to do that)

[Zema Bus]

Some of the Windows fanboys are gloating over this now, claiming this as proof that open source is "inherently insecure". But they're ignoring the fact that it could just as easily happen to Windows via a rouge MS employee who hired on for the sole purpose of doing the same thing, building trust until right time to deploy something. And in that case there would have been far fewer eyes looking at the closed source code for a random package.

[Grogan]

That was caught fast, considering there was no actual payload until very recently. Also, only silly distros would have upgraded xz that quickly, nothing "enterprise" or even mainstream like Ubuntu. It's also just one library, with a specific backdoor, yet I saw sensational headlines like "miscreant hacker almost takes over the whole internet"

Who says there aren't deliberate backdoors in Microsoft software for whatever reason? (e.g. copyright policing or aiding security agencies etc.) Did these Windows fanboys audit the source code? No? Oh, that's because they can't, even if they understood it.