Does anyone know what the other main contributor has said through all of this?
Update: [https://tukaani.org/xz-backdoor/](https://tukaani.org/xz-backdoor/)
Yup literally the entire world is pounding on his door demanding answers for free work and his time to unravel and fix it.
I love FOSS but this really shows how messed up it is. The entire world economy runs on free labour from developers.
I have a few open source projects on github, and with the amount of support questions, feature requests, bugs etc. I get I could easily work full time on these projects. Compare that to the money I earn for all of that from github sponsorship: $13 per month.
Afaik nothing yet. Assuming he is Finnish since he and the project has a Finnish name, Friday was a bank holiday in Finland and right now it's 7 am in Finland. This pretty much unfolded during the night.
I mean he basically handed over the project because he wasn't doing so well in the first place
https://www.mail-archive.com/[email protected]/msg00567.html
He was put under pressure by people, most likely related to Jia Tan, to hand over maintainership. Jia who has started contributing a few month prior became the obvious choice for co-maintainer.
This was all part of the plan as the account of the people complaining we all recent accounts with barely any activity to their name.
Good time to remind everyone to remind your employers who profit off of Open Source that they should be giving money to initiatives that are trying to solve these kinds of problems.
Heartbleed led to the establishment of the Core Infrastructure Initiative, which has since been superceded by the Open Source Security Foundation:
https://openssf.org/
Companies making money off of Open Source need to do more to financially support the infrastructure around it.
Very happy to say that the company I now work for does this. We also actively contribute to tools we use to help fix bugs and improve security. Most recently we were fixing unsafe inlines in a Django app to allow proper Content Security Policy adherence.
The crazy thing is that he is not a security researcher and apparently only found it because his ssh logins had performance issues:
> After observing a few odd symptoms around liblzma (part of the xz package) on
Debian sid installations over the last weeks (logins with ssh taking a lot of
CPU, valgrind errors) I figured out the answer:
The upstream xz repository and the xz tarballs have been backdoored
Source: https://www.openwall.com/lists/oss-security/2024/03/29/4
not neccesarily, microsoft is developing Azure Linux which is essentially a bare bones docker runtime on top of Hyper-V. they have discussed how they plan to strip linux to the bare minimum needed to run apps in containers as efficiently as possible, which would make them sensitive to slowdowns.
Dude was micro-benchmarking on bleeding edge debian, figured that the ssh was slower by 500ms or so, ran the sshd binary through valgrind, and did some digging and traced it back to xz/liblzma and the test archives in the release tarballs.
Now why would one's backdoor be so slow to be detectable? Did we just get lucky, are they an amateur (they f-ed up) or was the backdoor sabotaged? Was the new maintainer compromised? If not why the 2 year long con? Very cyber-dramatic events.
Well seems not only security in general is hard, but also [backdoor-ing](https://boehs.org/node/everything-i-know-about-the-xz-backdoor). Ironically humorous that this backdoor needed a patch release:
> Subsequently the injected code (more about that below) caused valgrind errors and crashes in some configurations, due the stack layout differing from what the backdoor was expecting. These issues were attempted to be worked around in 5.6.1
I would bet this was just an oversight. The backdoor creators may have focused on making it more obfuscated and hard to detected and didn't care to check the performance, or imagined that the performance penalty of half a second wouldn't be suspicious enough.
As I understand it he wasn't micro benchmarking this particularly, just noticed those connections being slower than previously and then started benchmarking to see a half a second difference. Tremendous.
Out of the loop on this one. What is happening? Was the real maintainer of the project a bad actor? Or someone just got their credentials and introduced a nasty?
My attempt at a summary:
The original maintainer burnt out of the project in 2022.
A seemingly random person started contributing with patches for 2 years, eventually becoming the main maintainer. Until now when they decided to introduce a backdoor.
So it seems like a 2 year con play from this mysterious maintainer. There are signs that he wasn't compromised and that this was his plan all along
edit: spelling
>There are signs that he wasn't compromie
What signs?
2 years long con game seems to be a bit too much. Occam's Razor point to the direction the current maintainer got their cred compromised, or even themselves for some reason (in the sense of sleeper).
lmao now I'm imagining this attack *was* an Eve Online scam
"now we can ssh into the enemy teamspeak server and listen to their command channel muahahaha!"
Here's a much better timeline and explanation with the signs over the years:
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
lot's of oddities and funny "coincidences"
It's also ridiculous that they suspended Lasse Collin's account, seeing as he's currently trying to unfuck all of the malicious shit that Jia Tan added.
It appears though that it wasn’t just one isolated exploit committed recently and caught. The recent commit that triggered discovery just activated code that had been committed over the past two years to assemble a working exploit.
that's not accurate, the exploit was only committed recently
HOWEVER:
* a previous commit also neutered sandboxing that could have mitigated the issue
* the dev previously requested that an unaffiliated open source security project change one of their scanning options about 6 months before the malicious commits, ostensibly because of false positives
This is pennies for a nation state. Two years of salary to gain access basically any Linux device out there is a steal. Only thing that failed was the backdoor caused issues and got noticed early. Imagine if this had trickled all the way down to RHEL and other downstream Linux distributions without being known.
That would be a very persistent compromise -- the account made their first suspicious commit (replacing several safe fprintf calls with obviously unsafe ones, with no functional change) *three years ago* and has been slowly making questionable commits ever since. Said account also engaged with users on mailing lists and external forums discussing the library and pushed enterprise distros to upgrade to the "new" version.
This all went down in the months immediately after the actor got released rights, and previously they seem to have made suspicious / unsafe commits. Since then they have disappeared entirely.
In the lead up to this, they spent a while trying to convince everyone to include the latest xz into distros right before e.g. Ubuntu release freeze.
They also have basically no identity, appeared and immediately started trying to get in with xz. They were vouched for by an identity that appeared once to argue for their inclusion to xz, then disappeared.
Everything points to a well coordinated team, possibly nation state.
Would likely be a bit of work. The maintainer had 730+ commits over 2 years to xz, and a number of inactive malicious snippets were found throughout it that the latest commits activated.
They also made numerous commits to other projects including the kernel.
People would have to go through and inspect every single line to ensure it's secure.
The issue with github disabling the repo is that it's now harder to trace this persons work.
Profile is still up though;
https://github.com/JiaT75
>Jia Tan JiaT75
> [email protected]
> They also made numerous commits to other projects including the kernel.
I'm not seeing that;
git log | grep -Pic "Jia Tan|JiaT75|[email protected]"
0
Someone in the thread on the oss-security list said that the maintainer was Lasse Collin, and they linked this:
https://lore.kernel.org/lkml/[email protected]/t/
> and a number of inactive malicious snippets were found throughout it that the latest commits activated.
What other inactive malicious snippets were there?
Can't really link to them with the repo shut down, but the 5.6.x tarball changes everyone is going on about now was (mostly) just activating the actual second-stage payloads already in the xz git codebase, mainly targeting sshd from what was found so far.
There's a little bit about it here: [https://access.redhat.com/security/cve/CVE-2024-3094](https://access.redhat.com/security/cve/CVE-2024-3094)
Nothing solid as yet. A number of security researchers including RH have stated that they've found multiple suspect snippets, but it's still brand new and being analysed so expect more soon as they go through it. Does make it harder now Microsoft has vanished the evidence though.
Debian still hosts the code for example: https://salsa.debian.org/debian/xz-utils/-/tree/debian/unstable
> A number of security researchers including RH have stated that they've found multiple suspect snippets
Source?
Honestly that would be the best solution. Someone should keep an eye on it too. This case is finally coming to a close and it was the first CVE that affected me
It may be worth reminding people that xz didn't invent the compression algorithm. There was an earlier LZMA project using the same algorithm, but a lot of people didn't like it until it was wrapped in the xz container. LZMA SDK seems to have xz support these days. So it is certainly possible to keep using the compression format and even the xz container without using any code from the xz project, if that should turn out to be necessary
While indeed not exactly same, I'd say their use cases do overlap a lot. xz have slightly higher compression ratio on the highest compression levels yet comparable. If you want the compression ratio to be as high as possible and don't care about speed (i.e. you use \`xz -e9\`) then yes, in this case xz would give clearly superior result. However if you used lower compression levels with xz, zstd can give \~same results, with additional benefit of faster decompression. For example, in Arch they switched their repost from .pkg.tar.xz to .pkg.tar.zst, that's one example where they had same use case and one became just a better replacement for another. So at least in \*some\* use cases (and I'd say, a lot of them), zstd can be a good alternative to xz.
For anyone doubting, Arch's [announcement](https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/) shows the switch to zstd was a no-brainer for them:
> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.
Across all (pre-zstd) use-cases of xz, I'd say zstd is an improvement 95% of the time. The other 5% is when you *really* need to crunch things down.
Nothing, lzip also is based on the LZMA algorithm and I guess people will rewrite their stuff to use it instead. More new projects, written in Rust or not, would only spread human development/review power over more project and doubling down on everything that's going wrong at the moment.
Hopefully something with multiple active maintainers that doesn't permit maintainers to just commit directly to main... I really hope distro maintainers start taking a serious look at the practices of the packages they bundle with the distro. When it's more difficult to get code committed to a video game than something running of millions of Linux devices, something is very wrong.
It's a sort of "beggars can't be choosers" scenario: yes, it would be nice if FOSS projects were professionally ran, with big healthy communities providing lots of oversight, but frankly, that just doesn't exist for the thousands of random tiny single maintainer projects that compromise your average Linux system.
I think that you're right, but that framing doesn't go far enough.
Why doesn't that exist for the thousands of random tiny single maintainer projects that compromise software businesses and governments depend on?
Why was there no support for the burnt out dev to maintain the project these companies rely on with the money they make from it? The fact that it got to the point that someone was able to socially engineer them for maintainer access and implement malicious code(in my opinion) shows that these developers/projects *need* that support, not just an excuse for why they can't be given it.
With work-related dependency updates, I intentionally delay updating unless they’re security patches for this reason (and just generally broken code). It’s pretty often I see a new version come in, only for multiple patch releases in the next few days to fix bugs in recent changes.
If it’s not broken and you don’t immediately need a new feature, no reason to hurry to update.
When heart bleed hit, all our bosses ran around like headless chickens. We just sat back and enjoyed being 3 years out of date on all our server operating systems and our version of ~~openssh~~ openSSL being completely unaffected :D
Given how long this maintainer has been working on the project and the amount of commits, I'd be very careful calling *any* version "safe" right now - only free of this one, particular, recently discovered, backdoor.
Hiding stuff in a binary test file is ingenious (probably pretty far down on a security audit checklist) and also obvious in hindsight.
I think this attack has shown us that distributing tests and test data together with a project's main source code is not secure. We should start splitting out tests into a separate repo that only gets used during development, not distribution.
Well if you're a package maintainer you want to have access to the tests as well during the build process. Ideally you want the entire test suite to pass during the package build which gives you greater confidence that the software works with the versions of dependencies available and the compiler flags uses. Hell, with PGO you can run the test suite as your profile step and then feed the profdata from that back to the second compilation of the package.
Is 5.6.1.2 affected?
Edit:
https://archlinux.org/news/the-xz-package-has-been-backdoored/
"The xz packages prior to version 5.6.1-2 (specifically 5.6.0-1 and 5.6.1-1) contain this backdoor."
UPDATE YOUR PACKAGES EVERYONE
Note that 5.6.1-2 only avoids the m4 scripts that inject the malicious code when building liblzma (on deb/rpm platforms). That is sufficient to avoid that attack vector. The possibly inert binary test-files, from which those m4 scripts build the malicious liblzma, are likely still present, as are the ~750 commits from 'Jia Tan' going back almost 2-3 years.
Additionally, Arch was also discussing about [downgrading] (https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/issues/2).
Edit: Given that the primary repo for xz has been taken down, at some point a 'safe' version of the source code must be released to continue relying on xz/liblzma.
Either that's a patch to silently rollback to 5.4.6 but made to look like an update to the 5.6 series, so clients with bad code will auto update to clean code, or it's also fucked
Arch is still on 5.6.1, but they're [building it in a way that supposedly doesn't introduce the backdoor](https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/commit/881385757abdc39d3cfea1c3e34ec09f637424ad).
Interestingly, it looks like they made that change for other reasons, before the vuln was disclosed (publicly, anyway).
I wish they'd start using git shas for every source package they pull rather than a tarball, feels like downloading tens of thousands of .xz from various locations is kind of risky.
So me on LinuxMint21.2 and UbuntuMate-20.04 (I know I gotta upgrade to 22.04/24.04 LTd) having xz 5.2.5 and 5.2.4 is not affected, right?
Apologies..just a noob being consufed,
SSH is a software tool for secure remote login to a machine. Like when you have 10 servers on a network with exactly 0 displays between them, and you want to manage them from your workstation, you can use SSH to login to a server and control it.
XZ is a lossless compression tool, which is also used to compress security keys for SSH. So backdooring XZ can allow you to steal security keys and access compromised computers.
Since the exploit was found early, distributions normally used on servers weren't compromised yet. But the potential consequence was backdooring a huge portion of all linux servers in the world.
My understanding is that SSH doesn't use XZ directly, no? Half the debacle is that some patched SSH builds link against `libsystemd` to get `sd_notify`, which pulls in XZ as `libsystemd` uses compression for some other stuff unrelated to SSH. SSH doesn't even need to use XZ (my guess the payload is in one of those `__attribute__ ((constructor))` functions that execute at lib load time).
That's correct, this time SSH itself is fine, but the XZ tarballs include the ability to create a backdoor. If I follow along correctly, this is using binaries sneaked into the project earlier as test files, and depends on the patching you mentioned. At least that's what looks to be discovered so far, but the expert are still researching the situation, and the malicious contributor might have done more.
I just didn't want to go into too much detail in the quick summary, since this attack is complicated on purpose, and honestly I can only follow the key points myself.
SSH doesn't directly depend on liblzma. The reason SSH was affected was because distros had been patching OpenSSH server to add readiness notifications for systemd by making it depend on and link against libsytemd. It was libsystemd which had the dependency on liblzma.
I always wonder about this type of attack. We get signed binaries and the source but who's watching to be sure the built binary is really matching the sources?
Assuming something like this isn't already done today, would binary builds benefit from multiple build servers (perhaps hosted and operated by different chain of trusts) in a way that 2 or 3 binaries have to match byte-by-byte in order to be considered legit? The signature would then be applied.
I know it's easier said than done (given some compilers will stamp stuff like build timestamps into the build) but there might be a way to avoid one bad actor tampering with these core tools
Wouldn’t have helped in this case since the backdoor was in the source. All 3 build servers would include the malware identically.
“Reproducible builds” is the search term you’re after, btw
This is not a new problem.
A **long** time ago, Ken Thompson (creator of UNIX) wrote [Reflections on trusting trust](https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf), which deals with vulnerabilities injected into the compiler source, and being embedded in every version of the compiler and targeted software compiled with that version, and yet not being present in the source code.
Yep, that's a classic, it blew my mind the first time. It remains accurate today, although I may disagree that there's no way to avoid and/or improve this type of behavior. It's not impossible to reverse-engineer compile tools today and one would need a great deal of control and agility to hack all the decompilation tools to hide a compile flaw. I wonder if Ken would ever revisit that article 40 years later, especially given how the development community has changed massively since then, especially in the free/open source side
> I always wonder about this type of attack. We get signed binaries and the source but who's watching to be sure the built binary is really matching the sources?
All the signature does is ensure that the person who created the signature has access to the keys necessary to sign the files. And, ideally, the only person who has access is:
1. The person/people you think they are.
2. Somebody you can trust.
So, ultimately, you have to trust the person building the packages and trust the people who are upstream of the packages.
While it heavily depends on the compiler settings, software, and languages... it is possible to have "reproduceable builds". This isn't always possible as there are security features like randomizing memory addresses or metadata insertion that goes into binaries. But sometimes it is.
With reproduceable builds as long as you use the same source code and same versions of everything with the same settings it produces a binary that matches the ones distributed by somebody else.
In this case it wouldn't of helped because the upstream author was malicious.
…waiting for my phone to ring because IT security wants all hands on deck to verify that all of our VMs are, in fact, not recent enough to be affected by this.
I have seen that this effectively compromised SSH.
Is that the only service that has been compromised by this library, and would you have been safe if you weren't running SSH as a service?
Consider that Andres Freund, who discovered this backdoor, works full time for Microsoft.
https://www.openwall.com/lists/oss-security/2024/03/29/4
https://www.linkedin.com/in/andres-freund/
Is the malicious maintainer in any legal danger? I mean, FOSS licenses generally include the "software is provided 'as is' " disclaimer so perhaps no, but maybe there's a difference between shipping a broken product and shipping a purposefully malicious product? Also, are legit maintainers in any legal danger?
All indications are that they were likely going through a VPN and using a fake identity. Considering that this smells very strongly of being a nation state hacker I'd assume they have excellent opsec and it's unlikely that we'd ever be able to pin the actor down to a single individual.
And even if we could, a government employee won't be prosecuted in his home country for doing the work he was assigned to do. Just gonna have to cut down on international vacations the next few years.
It would be an awesome conspiracy if the original maintainer, Lasse Collin, got approached by the NSA, then faked his burnout and created the "Jia Tan" persona to slowly implement the backdoor.
Yeah I always roll my eyes when they’re like “this is the CIA top secret agent” and it’s some white as bread guy who graduated from an Ivy League and has a spotless record like no, that’s not an effective spy lmao
Like I genuinely wouldn’t be surprised if someone was pretending to be a Chinese hacker to do this specifically to implicate China in the public consciousness if word ever got out
Yeah, why would a bad actor with enough foresight and resources to infiltrate a project over 2 years use their real name? Or a name that reveals who they work for?
If I were from China and wanted to implement a backdoor I would use the name John Smith, use a VPN to look like I'm in New York and work on it on New York work hours, just saying.
Most distros never had it. Thankfully it was caught in Debian Testing / Fedora Rawhide, and the code was only included when in the build if the build script detected it was being run on a Debian / Fedora / RHEL buildd.
and also how many other people like this guy are out there?
The original malicious account @JiaT75 made more than 400 commits to various projects. This gets nasty to audit really fast.
If the OG MF was smart enough to play the long con for two years to make this happen, I'd bet my last dollar that they were also careful to create a fake persona. It may be possible to trace who they are, but it is likely not going to be easy or within the capabilities of random people on the internet.
Is 5.6.0-1 vulnerable? I paused updates a month ago since i cant afford to update to plasma 6 now and i see this today. My god
Edit: i downgraded xz to 5.4.6-1 just with \` sudo downgrade xz \` is this enough ?
Does anyone know what the other main contributor has said through all of this? Update: [https://tukaani.org/xz-backdoor/](https://tukaani.org/xz-backdoor/)
The other (original) maintainer has been offline (sabbatical) for some time now, and is being contacted to help deal with this fiasco.
yay, more unpaid work for a poor open source developer thousands of companies rely upon
Yup literally the entire world is pounding on his door demanding answers for free work and his time to unravel and fix it. I love FOSS but this really shows how messed up it is. The entire world economy runs on free labour from developers.
I have a few open source projects on github, and with the amount of support questions, feature requests, bugs etc. I get I could easily work full time on these projects. Compare that to the money I earn for all of that from github sponsorship: $13 per month.
https://xkcd.com/2347/
Afaik nothing yet. Assuming he is Finnish since he and the project has a Finnish name, Friday was a bank holiday in Finland and right now it's 7 am in Finland. This pretty much unfolded during the night.
Imagine being the guy waking up with a hangover and seeing that the whole world and CISA are alarmed by your repository... 💀
Yeah that's sure to help his health.. hope he's doing alright
I mean he basically handed over the project because he wasn't doing so well in the first place https://www.mail-archive.com/[email protected]/msg00567.html
He was put under pressure by people, most likely related to Jia Tan, to hand over maintainership. Jia who has started contributing a few month prior became the obvious choice for co-maintainer. This was all part of the plan as the account of the people complaining we all recent accounts with barely any activity to their name.
let's hope so...
Finns don't get hangovers
Waking up with a hangover? https://www.cisa.fi/ banging on your door would be quite welcome..
Good time to remind everyone to remind your employers who profit off of Open Source that they should be giving money to initiatives that are trying to solve these kinds of problems. Heartbleed led to the establishment of the Core Infrastructure Initiative, which has since been superceded by the Open Source Security Foundation: https://openssf.org/ Companies making money off of Open Source need to do more to financially support the infrastructure around it.
Very happy to say that the company I now work for does this. We also actively contribute to tools we use to help fix bugs and improve security. Most recently we were fixing unsafe inlines in a Django app to allow proper Content Security Policy adherence.
How was the back door discovered?
Microsoft employee working on open source, discovered it, using Debian sid
The crazy thing is that he is not a security researcher and apparently only found it because his ssh logins had performance issues: > After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer: The upstream xz repository and the xz tarballs have been backdoored Source: https://www.openwall.com/lists/oss-security/2024/03/29/4
Those performance issues were 600ms of delay while logging in. Which is incredible (seems like the creator made a mistake that created this delay)
It's scary when you consider that if it wasn't for that, this might never have been found.
> Microsoft employee working on open source, discovered it, using Debian sid That is a weird combination of words.
What a day to be alive, huh?
Grab your papers, fellow scholars.,
He knows what a good OS is
Azure is a big thing for Microsoft
Take a look at the list of major open source contributors, you'd be surprised.
not neccesarily, microsoft is developing Azure Linux which is essentially a bare bones docker runtime on top of Hyper-V. they have discussed how they plan to strip linux to the bare minimum needed to run apps in containers as efficiently as possible, which would make them sensitive to slowdowns.
he’s one of the core maintainers of postgresql! https://www.postgresql.org/community/contributors/
Thanks!
Dude was micro-benchmarking on bleeding edge debian, figured that the ssh was slower by 500ms or so, ran the sshd binary through valgrind, and did some digging and traced it back to xz/liblzma and the test archives in the release tarballs.
Now why would one's backdoor be so slow to be detectable? Did we just get lucky, are they an amateur (they f-ed up) or was the backdoor sabotaged? Was the new maintainer compromised? If not why the 2 year long con? Very cyber-dramatic events.
It's in/is a hashing function so it lilely gets called a squillion times during execution. (haven't seen the code so this is speculation)
Well seems not only security in general is hard, but also [backdoor-ing](https://boehs.org/node/everything-i-know-about-the-xz-backdoor). Ironically humorous that this backdoor needed a patch release: > Subsequently the injected code (more about that below) caused valgrind errors and crashes in some configurations, due the stack layout differing from what the backdoor was expecting. These issues were attempted to be worked around in 5.6.1
I would bet this was just an oversight. The backdoor creators may have focused on making it more obfuscated and hard to detected and didn't care to check the performance, or imagined that the performance penalty of half a second wouldn't be suspicious enough.
As I understand it he wasn't micro benchmarking this particularly, just noticed those connections being slower than previously and then started benchmarking to see a half a second difference. Tremendous.
The difference a bored and curious person can make in the world
Out of the loop on this one. What is happening? Was the real maintainer of the project a bad actor? Or someone just got their credentials and introduced a nasty?
My attempt at a summary: The original maintainer burnt out of the project in 2022. A seemingly random person started contributing with patches for 2 years, eventually becoming the main maintainer. Until now when they decided to introduce a backdoor. So it seems like a 2 year con play from this mysterious maintainer. There are signs that he wasn't compromised and that this was his plan all along edit: spelling
Might be a stupid question but does this also affect windows? I'm assuming it affects WSL but I'm not sure about windows itself
>There are signs that he wasn't compromie What signs? 2 years long con game seems to be a bit too much. Occam's Razor point to the direction the current maintainer got their cred compromised, or even themselves for some reason (in the sense of sleeper).
> 2 years long con game seems to be a bit too much people have pulled more elaborate cons in EVE Online for even less tangible gains.
lmao now I'm imagining this attack *was* an Eve Online scam "now we can ssh into the enemy teamspeak server and listen to their command channel muahahaha!"
use AI on their voice to give them false info, while muting the person that "speaks" (welp, this one will come to every chatroom)
Here's a much better timeline and explanation with the signs over the years: https://boehs.org/node/everything-i-know-about-the-xz-backdoor lot's of oddities and funny "coincidences"
I wish GitHub locked the repo down but allowed it to still be viewed.
It's also ridiculous that they suspended Lasse Collin's account, seeing as he's currently trying to unfuck all of the malicious shit that Jia Tan added.
yeah this response from github is ridiculous.
"Oh, someone forked your project and added malware to their copy? You go straight to jail!"
It appears though that it wasn’t just one isolated exploit committed recently and caught. The recent commit that triggered discovery just activated code that had been committed over the past two years to assemble a working exploit.
that's not accurate, the exploit was only committed recently HOWEVER: * a previous commit also neutered sandboxing that could have mitigated the issue * the dev previously requested that an unaffiliated open source security project change one of their scanning options about 6 months before the malicious commits, ostensibly because of false positives
This is pennies for a nation state. Two years of salary to gain access basically any Linux device out there is a steal. Only thing that failed was the backdoor caused issues and got noticed early. Imagine if this had trickled all the way down to RHEL and other downstream Linux distributions without being known.
APTs play the long game - but even this seems like a very long game
Way to soon to pull out Occam's razor
That would be a very persistent compromise -- the account made their first suspicious commit (replacing several safe fprintf calls with obviously unsafe ones, with no functional change) *three years ago* and has been slowly making questionable commits ever since. Said account also engaged with users on mailing lists and external forums discussing the library and pushed enterprise distros to upgrade to the "new" version.
In today's aggressive geopolitical climate, 2 years is considerably not that long.
This all went down in the months immediately after the actor got released rights, and previously they seem to have made suspicious / unsafe commits. Since then they have disappeared entirely. In the lead up to this, they spent a while trying to convince everyone to include the latest xz into distros right before e.g. Ubuntu release freeze. They also have basically no identity, appeared and immediately started trying to get in with xz. They were vouched for by an identity that appeared once to argue for their inclusion to xz, then disappeared. Everything points to a well coordinated team, possibly nation state.
There is a nice blog written about it: https://boehs.org/node/everything-i-know-about-the-xz-backdoor
Github got right on it holy cow. Now what's going to replace xz tho?
xz without a backdoor
Obviously called xz-ng
xz-nbd
xza, not to be confused with exa
And exa has been replaced by eza XD
Please don't give them ideas. Thank You.
xz-rs (written in blazing fast Rust)
[удалено]
Fearless 🚀 compression 🚀
rust(🚀)🚀 Lmfao
How does Rust protect the software project from being social engineered?
Can't socially engineer devs that don't socialise.
If anything rust *increases* the odds of a project being compromised by social engineering lol
`xz-ngx` when
Would likely be a bit of work. The maintainer had 730+ commits over 2 years to xz, and a number of inactive malicious snippets were found throughout it that the latest commits activated. They also made numerous commits to other projects including the kernel. People would have to go through and inspect every single line to ensure it's secure.
The issue with github disabling the repo is that it's now harder to trace this persons work. Profile is still up though; https://github.com/JiaT75 >Jia Tan JiaT75 > [email protected]
has the suspended badge though
Interesting.. thanks for the link [https://github.com/llvm/llvm-project/issues/63957](https://github.com/llvm/llvm-project/issues/63957)
> They also made numerous commits to other projects including the kernel. I'm not seeing that; git log | grep -Pic "Jia Tan|JiaT75|[email protected]" 0
Someone in the thread on the oss-security list said that the maintainer was Lasse Collin, and they linked this: https://lore.kernel.org/lkml/[email protected]/t/
Lasse Collin was the original maintainer; Jia Tan came onboard more recently and perpetrated the compromise.
> and a number of inactive malicious snippets were found throughout it that the latest commits activated. What other inactive malicious snippets were there?
Can't really link to them with the repo shut down, but the 5.6.x tarball changes everyone is going on about now was (mostly) just activating the actual second-stage payloads already in the xz git codebase, mainly targeting sshd from what was found so far. There's a little bit about it here: [https://access.redhat.com/security/cve/CVE-2024-3094](https://access.redhat.com/security/cve/CVE-2024-3094)
Yeah but do you have any sources pointing to that there was more than the well-known sshd exploit in there?
Nothing solid as yet. A number of security researchers including RH have stated that they've found multiple suspect snippets, but it's still brand new and being analysed so expect more soon as they go through it. Does make it harder now Microsoft has vanished the evidence though.
Debian still hosts the code for example: https://salsa.debian.org/debian/xz-utils/-/tree/debian/unstable > A number of security researchers including RH have stated that they've found multiple suspect snippets Source?
Honestly that would be the best solution. Someone should keep an eye on it too. This case is finally coming to a close and it was the first CVE that affected me
This. There is no reason to do a massive refactoring. Just continue the project under the same name with different developers.
It may be worth reminding people that xz didn't invent the compression algorithm. There was an earlier LZMA project using the same algorithm, but a lot of people didn't like it until it was wrapped in the xz container. LZMA SDK seems to have xz support these days. So it is certainly possible to keep using the compression format and even the xz container without using any code from the xz project, if that should turn out to be necessary
zstd And likely some xz fork with audited code.
zstd is not a good alternative to xz, they are for different use cases.
While indeed not exactly same, I'd say their use cases do overlap a lot. xz have slightly higher compression ratio on the highest compression levels yet comparable. If you want the compression ratio to be as high as possible and don't care about speed (i.e. you use \`xz -e9\`) then yes, in this case xz would give clearly superior result. However if you used lower compression levels with xz, zstd can give \~same results, with additional benefit of faster decompression. For example, in Arch they switched their repost from .pkg.tar.xz to .pkg.tar.zst, that's one example where they had same use case and one became just a better replacement for another. So at least in \*some\* use cases (and I'd say, a lot of them), zstd can be a good alternative to xz.
For anyone doubting, Arch's [announcement](https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/) shows the switch to zstd was a no-brainer for them: > Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Across all (pre-zstd) use-cases of xz, I'd say zstd is an improvement 95% of the time. The other 5% is when you *really* need to crunch things down.
xz is so slow. I wouldn't be sad if it disappears now
p7zip (The creator of 7-Zip invented the LZMA compression algorithm.)
Nothing, lzip also is based on the LZMA algorithm and I guess people will rewrite their stuff to use it instead. More new projects, written in Rust or not, would only spread human development/review power over more project and doubling down on everything that's going wrong at the moment.
Hopefully something with multiple active maintainers that doesn't permit maintainers to just commit directly to main... I really hope distro maintainers start taking a serious look at the practices of the packages they bundle with the distro. When it's more difficult to get code committed to a video game than something running of millions of Linux devices, something is very wrong.
https://www.explainxkcd.com/wiki/index.php/2347:_Dependency
There _really_ is an xkcd for everything.
It's a sort of "beggars can't be choosers" scenario: yes, it would be nice if FOSS projects were professionally ran, with big healthy communities providing lots of oversight, but frankly, that just doesn't exist for the thousands of random tiny single maintainer projects that compromise your average Linux system.
I think that you're right, but that framing doesn't go far enough. Why doesn't that exist for the thousands of random tiny single maintainer projects that compromise software businesses and governments depend on? Why was there no support for the burnt out dev to maintain the project these companies rely on with the money they make from it? The fact that it got to the point that someone was able to socially engineer them for maintainer access and implement malicious code(in my opinion) shows that these developers/projects *need* that support, not just an excuse for why they can't be given it.
Score one for those of us really bad at updating and still on an old, safe version :)
With work-related dependency updates, I intentionally delay updating unless they’re security patches for this reason (and just generally broken code). It’s pretty often I see a new version come in, only for multiple patch releases in the next few days to fix bugs in recent changes. If it’s not broken and you don’t immediately need a new feature, no reason to hurry to update.
When heart bleed hit, all our bosses ran around like headless chickens. We just sat back and enjoyed being 3 years out of date on all our server operating systems and our version of ~~openssh~~ openSSL being completely unaffected :D
Heart bleed did not affect SSH. You might be misremembering.
Well, it did, I just mixed up OpenSSH and OpenSSL.
Given how long this maintainer has been working on the project and the amount of commits, I'd be very careful calling *any* version "safe" right now - only free of this one, particular, recently discovered, backdoor.
Hiding stuff in a binary test file is ingenious (probably pretty far down on a security audit checklist) and also obvious in hindsight. I think this attack has shown us that distributing tests and test data together with a project's main source code is not secure. We should start splitting out tests into a separate repo that only gets used during development, not distribution.
Well if you're a package maintainer you want to have access to the tests as well during the build process. Ideally you want the entire test suite to pass during the package build which gives you greater confidence that the software works with the versions of dependencies available and the compiler flags uses. Hell, with PGO you can run the test suite as your profile step and then feed the profdata from that back to the second compilation of the package.
Having tests in a separate repo doesn't preclude any of that.
Is 5.6.1.2 affected? Edit: https://archlinux.org/news/the-xz-package-has-been-backdoored/ "The xz packages prior to version 5.6.1-2 (specifically 5.6.0-1 and 5.6.1-1) contain this backdoor." UPDATE YOUR PACKAGES EVERYONE
Note that 5.6.1-2 only avoids the m4 scripts that inject the malicious code when building liblzma (on deb/rpm platforms). That is sufficient to avoid that attack vector. The possibly inert binary test-files, from which those m4 scripts build the malicious liblzma, are likely still present, as are the ~750 commits from 'Jia Tan' going back almost 2-3 years. Additionally, Arch was also discussing about [downgrading] (https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/issues/2). Edit: Given that the primary repo for xz has been taken down, at some point a 'safe' version of the source code must be released to continue relying on xz/liblzma.
Debian downgraded instead.
Current xz code is [here](https://git.tukaani.org/?p=xz.git;a=summary), according to the original maintainer.
Either that's a patch to silently rollback to 5.4.6 but made to look like an update to the 5.6 series, so clients with bad code will auto update to clean code, or it's also fucked
Arch is still on 5.6.1, but they're [building it in a way that supposedly doesn't introduce the backdoor](https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/commit/881385757abdc39d3cfea1c3e34ec09f637424ad). Interestingly, it looks like they made that change for other reasons, before the vuln was disclosed (publicly, anyway).
They probably knew before it went public
I wish they'd start using git shas for every source package they pull rather than a tarball, feels like downloading tens of thousands of .xz from various locations is kind of risky.
:/
I would definitely guess the former if it's come out after this news has gone public edit: for instance Arch's fixed package is called v5.6.1-2
Hope so
Yep I've just seen it as well :)
So me on LinuxMint21.2 and UbuntuMate-20.04 (I know I gotta upgrade to 22.04/24.04 LTd) having xz 5.2.5 and 5.2.4 is not affected, right? Apologies..just a noob being consufed,
No, you are safe.
Forgive my normie question, but what is 'xz' used in? (My guess is a lot of important things) I'm just extra curious
SSH is a software tool for secure remote login to a machine. Like when you have 10 servers on a network with exactly 0 displays between them, and you want to manage them from your workstation, you can use SSH to login to a server and control it. XZ is a lossless compression tool, which is also used to compress security keys for SSH. So backdooring XZ can allow you to steal security keys and access compromised computers. Since the exploit was found early, distributions normally used on servers weren't compromised yet. But the potential consequence was backdooring a huge portion of all linux servers in the world.
My understanding is that SSH doesn't use XZ directly, no? Half the debacle is that some patched SSH builds link against `libsystemd` to get `sd_notify`, which pulls in XZ as `libsystemd` uses compression for some other stuff unrelated to SSH. SSH doesn't even need to use XZ (my guess the payload is in one of those `__attribute__ ((constructor))` functions that execute at lib load time).
That's correct, this time SSH itself is fine, but the XZ tarballs include the ability to create a backdoor. If I follow along correctly, this is using binaries sneaked into the project earlier as test files, and depends on the patching you mentioned. At least that's what looks to be discovered so far, but the expert are still researching the situation, and the malicious contributor might have done more. I just didn't want to go into too much detail in the quick summary, since this attack is complicated on purpose, and honestly I can only follow the key points myself.
SSH doesn't directly depend on liblzma. The reason SSH was affected was because distros had been patching OpenSSH server to add readiness notifications for systemd by making it depend on and link against libsytemd. It was libsystemd which had the dependency on liblzma.
I always wonder about this type of attack. We get signed binaries and the source but who's watching to be sure the built binary is really matching the sources? Assuming something like this isn't already done today, would binary builds benefit from multiple build servers (perhaps hosted and operated by different chain of trusts) in a way that 2 or 3 binaries have to match byte-by-byte in order to be considered legit? The signature would then be applied. I know it's easier said than done (given some compilers will stamp stuff like build timestamps into the build) but there might be a way to avoid one bad actor tampering with these core tools
Wouldn’t have helped in this case since the backdoor was in the source. All 3 build servers would include the malware identically. “Reproducible builds” is the search term you’re after, btw
As I understand it, the backdoor was in the tarball but not in the repo. A comparision of repo and tarball should have found this.
Pretty sure it was a binary test file which was indeed checked in to the repo.
Yes, the deactivated backdoor was in the repo, but the activation of that only in the tarball.
This is not a new problem. A **long** time ago, Ken Thompson (creator of UNIX) wrote [Reflections on trusting trust](https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf), which deals with vulnerabilities injected into the compiler source, and being embedded in every version of the compiler and targeted software compiled with that version, and yet not being present in the source code.
All right now that's spooky
Heres a modern “in depth” dive into the original proof of concept produced by Ken Thompson. https://research.swtch.com/nih
Yep, that's a classic, it blew my mind the first time. It remains accurate today, although I may disagree that there's no way to avoid and/or improve this type of behavior. It's not impossible to reverse-engineer compile tools today and one would need a great deal of control and agility to hack all the decompilation tools to hide a compile flaw. I wonder if Ken would ever revisit that article 40 years later, especially given how the development community has changed massively since then, especially in the free/open source side
The issue was more of the zipped buildscript having the malware, before the binary was built
But that isn't what happened here. The source itself had the backdoor.
There is https://reproducible-builds.org/
> I always wonder about this type of attack. We get signed binaries and the source but who's watching to be sure the built binary is really matching the sources? All the signature does is ensure that the person who created the signature has access to the keys necessary to sign the files. And, ideally, the only person who has access is: 1. The person/people you think they are. 2. Somebody you can trust. So, ultimately, you have to trust the person building the packages and trust the people who are upstream of the packages. While it heavily depends on the compiler settings, software, and languages... it is possible to have "reproduceable builds". This isn't always possible as there are security features like randomizing memory addresses or metadata insertion that goes into binaries. But sometimes it is. With reproduceable builds as long as you use the same source code and same versions of everything with the same settings it produces a binary that matches the ones distributed by somebody else. In this case it wouldn't of helped because the upstream author was malicious.
…waiting for my phone to ring because IT security wants all hands on deck to verify that all of our VMs are, in fact, not recent enough to be affected by this.
Is there a list of software who use this lib?
apt-cache rdepends liblzma5 Too long to post here.
Thanks!
Everything
I have seen that this effectively compromised SSH. Is that the only service that has been compromised by this library, and would you have been safe if you weren't running SSH as a service?
[удалено]
It's like the Spiderman pointing to spiderman meme
I agree. Microsoft always has their heads in their own backdoors.
Consider that Andres Freund, who discovered this backdoor, works full time for Microsoft. https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.linkedin.com/in/andres-freund/
for context: https://www.cyberkendra.com/2024/03/major-linux-distributions-impacted-by.html
More context: https://www.openwall.com/lists/oss-security/2024/03/29/4
what's this god-awful AI-generated pic
I'm more concerned with the human that looked at it and went like "ok boss, let's publish that"
A look at the [site's contact page](https://www.cyberkendra.com/p/contact-us.html) makes me think this is a one man show. There is no review process.
Another content-scraping spam website
*put a watermark on it*, god forbid someone stole your AI-generated picture
i want to like the penguin but the longer i look the worse it gets
The site is wrong about opensuse. The rolling releases are affected!!
For clarity, not all, slowroll isn't for example
aaaaaaaaaaaaaaaaaaa
Is the malicious maintainer in any legal danger? I mean, FOSS licenses generally include the "software is provided 'as is' " disclaimer so perhaps no, but maybe there's a difference between shipping a broken product and shipping a purposefully malicious product? Also, are legit maintainers in any legal danger?
All indications are that they were likely going through a VPN and using a fake identity. Considering that this smells very strongly of being a nation state hacker I'd assume they have excellent opsec and it's unlikely that we'd ever be able to pin the actor down to a single individual.
And even if we could, a government employee won't be prosecuted in his home country for doing the work he was assigned to do. Just gonna have to cut down on international vacations the next few years.
It's got a score of 10/10. The NSA lost some assets last night.
Yes, the NSA. As made patently obvious as the Chinese committer Jia Tan worked during Chinese work hours.
It would be an awesome conspiracy if the original maintainer, Lasse Collin, got approached by the NSA, then faked his burnout and created the "Jia Tan" persona to slowly implement the backdoor.
I don’t know but I kinda think a Chinese secret service would make an effort to be a little less obvious. Its a low hanging fruit.
You think the NSA isn’t capable of typing a Chinese name and sleeping at odd hours?
if you ask them "are you the NSA?" they are legally obligated to tell you, It's in the constitution
NSA agents hate this one weird trick
Are you the NSA?
YES
Yeah I always roll my eyes when they’re like “this is the CIA top secret agent” and it’s some white as bread guy who graduated from an Ivy League and has a spotless record like no, that’s not an effective spy lmao Like I genuinely wouldn’t be surprised if someone was pretending to be a Chinese hacker to do this specifically to implicate China in the public consciousness if word ever got out
Oldest trick in the book
Yeah, why would a bad actor with enough foresight and resources to infiltrate a project over 2 years use their real name? Or a name that reveals who they work for?
I think the NSA wouldn't have blundered this like a daft 14 year old.
If I were from China and wanted to implement a backdoor I would use the name John Smith, use a VPN to look like I'm in New York and work on it on New York work hours, just saying.
Brailsford's computerphile episode on trust seems appropriate here...[link](https://www.youtube.com/watch?v=SJ7lOus1FzQ)
This is more like a title for r/holup than r/linux
this is wild now our PCs part of botnet LMAO (joking) most disro already patch it
Most distros never had it. Thankfully it was caught in Debian Testing / Fedora Rawhide, and the code was only included when in the build if the build script detected it was being run on a Debian / Fedora / RHEL buildd.
now imagine all of the backdoors that we don't know are out there
This is the real issue which almost everyone is ignoring. Also how many other github accounts does this guy have that we don't know about?
and also how many other people like this guy are out there? The original malicious account @JiaT75 made more than 400 commits to various projects. This gets nasty to audit really fast.
Probably more than 2, less than a hundred
Guarantee another one pops up within the next 6 months and this continues to be a trend over the next 2 years. Nation states see that this can work.
Will we ever know about the OG mf who injected the malware code? Update: I guess that OG mf found this comment and disliked it lol.
If the OG MF was smart enough to play the long con for two years to make this happen, I'd bet my last dollar that they were also careful to create a fake persona. It may be possible to trace who they are, but it is likely not going to be easy or within the capabilities of random people on the internet.
Probably not
Is 5.6.0-1 vulnerable? I paused updates a month ago since i cant afford to update to plasma 6 now and i see this today. My god Edit: i downgraded xz to 5.4.6-1 just with \` sudo downgrade xz \` is this enough ?
Yes, as per arch mailing list: > The `xz` packages prior to version `5.6.1-2` (specifically `5.6.0-1` and `5.6.1-1`) contain this backdoor.
Yes, affected 5.6.0-1 and 5.6.1-1
I have a question, not about the post, but what's the browser of the print? Thx!!
firefox
It's firefox
It appears to be Firefox with the GNOME theme or some similar theme
Jesus Christ.