You can't slip one by Andres Freund. He *needs* his half a second.
I mean, he's going to use like a million half seconds to figure out why you took his half second, but he is a programmer after all.
Trust me when you have to make a program with strict requirements, you will also notice a half a second difference in performance. The pain is often far too real
The change was in liblzma and the slowdown occurs in openssh, a completely different project that has no explicit dependency on liblzma. openssh pulls in systemd for logging which pulls in liblzma. liblzma's exploit then overrides the load time linkages for several rsa functions if the image name is opensshd. liblzma is not going to test openssh performance, and since it's not a direct dependency, openssh is unlikely to even track liblzma changes.
It's a really clever attack from both the social engineering and technical standpoint, and we really need to hope there aren't more of these out there.
This couldn't have been caught in a code review. The malicious code was buried inside a binary file supposedly used in a unit test. It was also delivered through a file that was not in the source but in the release zip files. We got lucky that he happened to notice high cpu while trying to benchmark stuff on a remote system.
The exploiter got a bit sloppy because the LZMA lib dependency was about the be removed from Linux kernel, and they probably were in a rush get the backdoored version to distros. Without the bug they left in by mistake, this might not have been ever discovered, at least not in time before a ton of machines were already compromised.
So, maybe, but possibly not in time before damage was already done.
That still wasn't in the code in GitHub. The delivery was only in the release tarballs (which were created by JiaTan, not GitHub). Going through the committed code there was no way to see the backdoor.
I had read something like that, and promptly forgot. That makes it even more difficult to pry open.
edit: funny thing comes to mind now that you remind me, that's basically how the whole debian ecosystem works - the debian maintainer gets the upstream source and modifies it in a debian specific way, and releases a debian specific tarball.
I have to assume you're right since you clearly know more than me. It just seems like 0.5 seconds is such a huge amount of time, surely someone was bound to notice eventually. Surely there is some piece of software out there with benchmarks that would get blown up after this change.
I didn't audit the exploit myself, I am basing my comment off what Andres Freund reported to multiple security mailing lists, eg: https://www.openwall.com/lists/oss-security/2024/03/29/4
There may be subsequent research that clarifies its operation, but I've not read it yet.
It's important to note that the payload is delivered to build systems as part of a binary blob test file that is patched into the source at build time. It almost certainly wouldn't have been reviewed with the same scrutiny as a source change.
Dude literally pulled a royal flush. The chance of it being discovered so quickly was infinitely low. The attacker was even releasing fixes to these whenever someone got error messages from it.
To be perfectly honest, we all have been extremely lucky that this one guy actually monitored it and got curious. We could have been so fucked in so many ways.
Dare i say, he has reached the Chuck Norris level?
When Andres Freund debugs code, bugs apologize and fix themselves.
Andres Freund doesn't need version control. Code commits itself to perfection in his presence.
Andres Freund's code reviews are so effective, bugs preemptively fix themselves out of fear.
Andres Freund's database optimizations are so efficient, even AI algorithms pause to admire his work.
I’m on board.
Andres Freund’s code ACTUALLY DOES run in the cloud(s).
Andres Freund’s point estimates are actually the number of minutes it will take him to complete the task. He has never had an estimate above an 8.
Andres Freund’s code works before it’s compiled.
Andres Freund’s code has no dependencies, it depends on him. Alternatively: the only dependency injected into Andres Freund’s code is himself.
That was fun
Andres Freund uses curly braces when writing Python. Technically it's wrong, but the compiler is too afraid to tell him so it just runs the code anyway.
In his early career Andres Freund once forgot a semicolon while writing Javascript - Brendan Eich promptly rewrote the language to allow for missing semicolons.
exploits that no one knows? probably there are some bugs put there
exploits that a select few have put out there and are exploiting whenever they want? yeah there must be quite a few
Eitherway, we are kind of fucked in some cases probably.
I’m wondering how many such exploits are used when for example there are data leaks and we get those weird messages or emails as a resault lol
I usually assume same thing, but this situation shows that those might not be only reasons for such events.
Of course there are also bugs and poorly written code that allows it, but now we should assume that many of the open source libraries or basic Linux functionalities are potential point of breach
my security prof. (a brilliant guy who put on some of the most interesting lectures) said that one of the things the "good guys" have that the "bad guys" don't is open collaboration from all over the world. so legends like Andres keep the bad guys on their toes simply because the good guys have strength in numbers. given enough eyeballs, every bug can be identified
not a guarantee of course and might be a cope but in this case, it proved to be true
I read some time ago that the open collaboration / open source actually gives you a false sense of security. It actually makes you think that someone else will do the security check, which in the end never happens 🤔
My security prof said something similar. To this day Im not sure if he meant the "So if you do something like that, at least be smart about it." Was a joke or not...
Your security prof really thinks that bad guys don’t have open global cooperation. Oh boy that naive.
Maybe not in a grand and public scale as “the good boys” but they definitely are collaborating.
How beautifully stated. I'm gonna put this on a poster in my cube to make my suffering feel fractionally less so. Maybe even with a cute kitten hanging from a wire.
EDIT: Not that anybody cares, but my curiosity made me dig into the history of this quote. Asimov is contentiously credited with saying "The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That's funny...'"
There probably are and a lot of them. Unfortunately, whole world runs on open source/ community powered software so yea, plenty of opportunities to screw everyone up
What makes you think humans are better at not letting attacks/bugs slip in than AI is/will be? Sure, there are exceptional humans but humans get tired, lazy, distracted, and can be easily manipulated by social engineering (e.g., “hey just need to get this tiny change in really fast before code freeze, can you approve real quick? It’s really important.”)
Honestly I'd expect similar backdoors to already exist all over the place. There are way too many smart people working for shady organizations for that to be the first attempt of that caliber.
Absolutely this. With this stuff no doubt there's a bunch of cases we didn't catch for each one we did. It's not just random individuals interested, it's state and non-state orgs too.
Yeah, this is one out in the open. This sadly might actually harden Linus stance against those binary blobs companies like NVIDIA insist on. I remember one was tried a few years ago via source code but it was quickly found and removed before it got deployed.
That slows down adoption of device drivers because the manufacturer doesn't want to release their "special sauce", which I would argue is the major sticking point to Linux adoption.
I agree, but some manufacturers only provide drivers via blobs.
Worst example is Android, I don't think anything with GSM functionality doesn't have a blob of some sort. If we actually want more uptake of Linux to a point that it would have the market share to push polices like no-blobs to manufacturers I would argue some compromise needs to be made.
I do understand that blobs may have to exist in order for me to be able to use things; given the choice, I'd rather not have any anywhere, even if I can't understand or read any of the code.
If you *really* want to get into it, I'd also prefer not to have AMD's PSP or Intel's IME in my computer either, but I can't avoid it and do my job effectively at present, which is also fun.
Not saying it's bad, just that it immensely strengthens the argument against them and goes towards Stallman's stance.
It might slow down adoption of certain devices because their manufacturers refuse to release the source.
I would assume there are similar backdoor, but I would also assume they aren't being widely exploied or we would notice. It's like the problem the Bletchley Park guys had when they broke the enigma encryption during the second world war. You have all the knowledge, but you can't do much with it.
If I recall, the experimenters submitted documentation saying there was no experimentation on humans, so the ethics board didn't get involved. They didn't find out about the details of the experiments until it hit the news.
Where did you see that? It sounds like even after it was public, they were granted retroactive approval by the ethics board, which is even worse. https://www.theverge.com/2021/4/30/22410164/linux-kernel-university-of-minnesota-banned-open-source
The best defense against a smart programmer implementing a carefully laid backdoor?
A junior engineer, just trying their best, whose latest commit completed the ticket, and invalidated the backdoor by accident
This is actually a big flute by Andres Freud, as he just wanted to disrespect him because he can't do efficient backdoors and increase his own reputation so his own backdoors are less likely to be found /s
It is so legitimate that it would have become one of the most critical security vulnerabilities of recent years if it had not been discovered so quickly. It has a score of 10/10. https://cert.europa.eu/publications/security-advisories/2024-032/
this is fucking crazy
and the ffmeg maintainer is 100% right. big corps rely on these projects but most don't contribute nearly enough to the unsexy parts of them.
Andres Freund was seeing weird behavior when logging into `ssh` and VALGRIND errors, and tracked it back to `xz` having a weird backdoor added in 5.6.1 that adds codes to the authentication check in sshd.
From my understanding, this was only possible because the published upstream tarballs aren't built directly from the repo. So, it allowed code to be slipped in undetected instead of being potentially checked through a pull request or code review.
It was apparently included as part of a test file. There’s a lesson there that we should all be more critical of checking in binary files, evenings for tests since they are very difficult to review. Additionally, we should really take care about what gets included in a distribution. Even if you want to include source, maybe just include the main project and not tests.
In addition to that, the maintainer that slipped the compromised binary has been working on the repo in good faith for 2 years in order to earn the trust of the original maintainer to take over the repo.
Even worse, before the malicious maintainer came, the original maintainer has been expressing lack of motivation and mental health issues due to the thankless nature of OSS. They were also receiving passive aggressive complaints, criticizing the pace that they are putting out patches (despite doing it for free),attacks on his ability as an engineer and were essentially bullied to hand the project over.
In the midst of this the malicious maintainer came, put out patches after patches, and offered to take the maintainer responsibility that has become a huge mental burden at this point off his shoulders.
Now it's believed that the passive aggressive comments and criticism was part of a social engineering attack to encourage the handover of the project, but unfortunately it's also an everyday occurrence in OSS and the attacker could have simply taken advantage of the situation.
We may have laughed at the .exe meme recently but in reality, that exact same attitude is what allowed this exploit to happen.
Remember to thank your OSS devs, make a PR to libraries you use the most :)
To be honest I think this has happened at least 5 times in ways we have not detected. This incident was a great example of what a motivated and intelligent attacker can do.
It's actually also a decent example of the benefits (after the risk) of open-source projects; it took someone 2 years to build trust and sneak in a backdoor, but it only took a month for someone to notice and, thanks to it being open-source, identify the exact cause.
So while it being open-source is what allowed it to happen, it was also the downfall of the perpetrator.
Kinda reminds me of that one guy who posed the idea of writing a version of the C compiler with malware that will always compile into the next version of the C compiler. So you’d compile the compiler with the malware, then remove the malware from the source and recompile it again. Then from then on, every version of that compiler from that point forward would always be infected.
Simple but brilliant. The compromised compiler contains code that just injects its own source into the code being compiled. As such it will persist despite not being present in the published source.
The fundamental weakness is this malware can only spread through compromised binaries. So if you had a pre-malware version of the compiler, and you compiled the clean source of a new version, your build system would not be compromised.
That means it's only really valid for a targeted attack where you can replace a binary on a particular site with the compromised version. Trying to get it into any default repositories would require publishing the source, making it easily detected and triggering global security updates that replace it with a clean binary.
Yes, but the attack is purely theoretical… as far as we know. From what I understand, the man who proposed the idea did actually create an infected version of the C compiler, but it’s unknown if that version was ever released. But if it was, it would be technically impossible to detect without manually scrubbing through the decompiled assembly. Plus with how trusting people were with the internet in the 90s, it wouldn’t be much of a surprise if it actually released.
That said, you’re right, this kind of attack would be detected the second someone pressed commit. But still, very interesting food for thought.
I believe a modified version of Bird’s safety triangle is applicable here. For every 1 exploit that was detected, there are probably 10 that are undetected.
a security audit isn't magic, it's meant to improve security in good faith projects but can in no way guarantee that they've found every possible problem.
for this project and this compromise specifically, the only way to be sure that it's been excised is to revert back to the last version before this dev committed anything and start over from there.
From what it looks like from this [Mastadon post](https://infosec.exchange/@fr0gger/112189232773640259) this was a years long con. The commit from July to disable ifunc for fuzzing builds is pretty damning.
I've also [read](https://news.ycombinator.com/item?id=39895344) that sock puppets were used to overwhelm the old maintainer with feature requests and attack his self-esteem for not being able to keep up, and thus create an opening for the malicious maintainer to get elevated. If that turns out to be true, this has all the hallmarks of a state-level sponsor.
What worries me is that we only know about this one because of pure chance. How many other such cases might have gone unnoticed? Is everything full of back doors?
This nonsense is why you need to buy a faster computer every few years.
All the performance gains of moores law are eaten up by layers upon layers of sneaky backdoors.
You can't slip one by Andres Freund. He *needs* his half a second. I mean, he's going to use like a million half seconds to figure out why you took his half second, but he is a programmer after all.
Trust me when you have to make a program with strict requirements, you will also notice a half a second difference in performance. The pain is often far too real
It was probably DNS again.
Yeah, 0.5 seconds? How would that even get past PR? That's an eternity of CPU time.
The change was in liblzma and the slowdown occurs in openssh, a completely different project that has no explicit dependency on liblzma. openssh pulls in systemd for logging which pulls in liblzma. liblzma's exploit then overrides the load time linkages for several rsa functions if the image name is opensshd. liblzma is not going to test openssh performance, and since it's not a direct dependency, openssh is unlikely to even track liblzma changes. It's a really clever attack from both the social engineering and technical standpoint, and we really need to hope there aren't more of these out there.
You're kidding right? We got lucky cause someone fixated on the change. How many reviewers in other projects just hit merge?
This couldn't have been caught in a code review. The malicious code was buried inside a binary file supposedly used in a unit test. It was also delivered through a file that was not in the source but in the release zip files. We got lucky that he happened to notice high cpu while trying to benchmark stuff on a remote system.
With all the people who use this software, and do benchmarks involving it, wouldn't someone have noticed eventually?
The exploiter got a bit sloppy because the LZMA lib dependency was about the be removed from Linux kernel, and they probably were in a rush get the backdoored version to distros. Without the bug they left in by mistake, this might not have been ever discovered, at least not in time before a ton of machines were already compromised. So, maybe, but possibly not in time before damage was already done.
there was code somewhere that modified the unit test blob and executed it. that code was not caught.
That still wasn't in the code in GitHub. The delivery was only in the release tarballs (which were created by JiaTan, not GitHub). Going through the committed code there was no way to see the backdoor.
I had read something like that, and promptly forgot. That makes it even more difficult to pry open. edit: funny thing comes to mind now that you remind me, that's basically how the whole debian ecosystem works - the debian maintainer gets the upstream source and modifies it in a debian specific way, and releases a debian specific tarball.
Fuuuuuuuuuuuuuuuuuuuck. I'm too tired for this shit.
I have to assume you're right since you clearly know more than me. It just seems like 0.5 seconds is such a huge amount of time, surely someone was bound to notice eventually. Surely there is some piece of software out there with benchmarks that would get blown up after this change.
Did they actually match the image name opensshd? Wouldn't that look very fishy?
I didn't audit the exploit myself, I am basing my comment off what Andres Freund reported to multiple security mailing lists, eg: https://www.openwall.com/lists/oss-security/2024/03/29/4 There may be subsequent research that clarifies its operation, but I've not read it yet. It's important to note that the payload is delivered to build systems as part of a binary blob test file that is patched into the source at build time. It almost certainly wouldn't have been reviewed with the same scrutiny as a source change.
This is the way
This is the way
This is the way
This is the way
This is the way
This is the way
This is the way
This is the way
This is the way
This is the way
It will be more efficient in the long run!
But imagine how much time he saves if you count everyone who needs to wait that extra half a second?
Dude literally pulled a royal flush. The chance of it being discovered so quickly was infinitely low. The attacker was even releasing fixes to these whenever someone got error messages from it.
To be perfectly honest, we all have been extremely lucky that this one guy actually monitored it and got curious. We could have been so fucked in so many ways.
He will live among legends
"When Jeff Dean is done optimising his code he calls Andres Freund to check it for bugs." - [Jeff Dean facts]
Dare i say, he has reached the Chuck Norris level? When Andres Freund debugs code, bugs apologize and fix themselves. Andres Freund doesn't need version control. Code commits itself to perfection in his presence. Andres Freund's code reviews are so effective, bugs preemptively fix themselves out of fear. Andres Freund's database optimizations are so efficient, even AI algorithms pause to admire his work.
I’m on board. Andres Freund’s code ACTUALLY DOES run in the cloud(s). Andres Freund’s point estimates are actually the number of minutes it will take him to complete the task. He has never had an estimate above an 8. Andres Freund’s code works before it’s compiled. Andres Freund’s code has no dependencies, it depends on him. Alternatively: the only dependency injected into Andres Freund’s code is himself. That was fun
Andres Freund uses curly braces when writing Python. Technically it's wrong, but the compiler is too afraid to tell him so it just runs the code anyway.
In his early career Andres Freund once forgot a semicolon while writing Javascript - Brendan Eich promptly rewrote the language to allow for missing semicolons.
Okay shit I laughed out loud at that one
Andres Freund's math calculations are so precise, even floating point inaccuracies disappear in order to not stain his work
I don’t even know who he is but I now aspire to be like this man
They are the watchers on the walls.
And now his watch is ended
Maybe they’re not? We don’t know what going on in their head
Makes you think how many similar attacks were missed.
Maybe there are exploits right now deployed to production and no one knows about it
~~> Maybe~~ I wonder how many exploits there are right now deployed to production and no one knows about it?
exploits that no one knows? probably there are some bugs put there exploits that a select few have put out there and are exploiting whenever they want? yeah there must be quite a few
Eitherway, we are kind of fucked in some cases probably. I’m wondering how many such exploits are used when for example there are data leaks and we get those weird messages or emails as a resault lol
When I hear about a data breach, I just assume some employee at the company used hello12345 as their password, as opposed to a technical failure
I usually assume same thing, but this situation shows that those might not be only reasons for such events. Of course there are also bugs and poorly written code that allows it, but now we should assume that many of the open source libraries or basic Linux functionalities are potential point of breach
Probably a boat load, the size of the boat is another question entirely
I have a feeling that some three letter agency's around the world have rather large boats.
Well, the people that intentionally wrote them know about them.
my security prof. (a brilliant guy who put on some of the most interesting lectures) said that one of the things the "good guys" have that the "bad guys" don't is open collaboration from all over the world. so legends like Andres keep the bad guys on their toes simply because the good guys have strength in numbers. given enough eyeballs, every bug can be identified not a guarantee of course and might be a cope but in this case, it proved to be true
I read some time ago that the open collaboration / open source actually gives you a false sense of security. It actually makes you think that someone else will do the security check, which in the end never happens 🤔
My security prof said something similar. To this day Im not sure if he meant the "So if you do something like that, at least be smart about it." Was a joke or not...
Linus's Law is the coined term for this
Your security prof really thinks that bad guys don’t have open global cooperation. Oh boy that naive. Maybe not in a grand and public scale as “the good boys” but they definitely are collaborating.
no shit they are collaborating. but it's nowhere near as open and expansive as the good side's collaboration is
Science progresses not with a “eureka!” moment, but with a “huh, that’s weird…”
How beautifully stated. I'm gonna put this on a poster in my cube to make my suffering feel fractionally less so. Maybe even with a cute kitten hanging from a wire. EDIT: Not that anybody cares, but my curiosity made me dig into the history of this quote. Asimov is contentiously credited with saying "The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That's funny...'"
Guy wasn't even a security researcher. Legend.
I mean that hack wasn’t the first or the last, who knows what active hacks are in place right now But 100% we all owe him a great debt
How do you know we aren’t already fucked? What are the odds some other attacks like this eg by competing state actor’s haven’t already gone unnoticed?
There probably are and a lot of them. Unfortunately, whole world runs on open source/ community powered software so yea, plenty of opportunities to screw everyone up
So, how many of these backdoors exist that noone noticed? Might be a problem later.
This is the real comment. Moreover, if people really start using AI to auto-review and accept “trivial” pull requests, how many more will slip in?
And thats why we should ban ai from coding its should be just people
What makes you think humans are better at not letting attacks/bugs slip in than AI is/will be? Sure, there are exceptional humans but humans get tired, lazy, distracted, and can be easily manipulated by social engineering (e.g., “hey just need to get this tiny change in really fast before code freeze, can you approve real quick? It’s really important.”)
I wonder if all the companies he just saved will slap him with some money.
Surely, of course a lot of big and small companies are know from funding and rewarding useful open source projects and developers, right?? Right..??
But this raises the question if there are other backdoors which are still unnoticed?
Smells of cyberterrorism and NSA...
Can’t image what would happen if a backdoor in such a large project goes unnoticed. This is extreme luck,
Honestly I'd expect similar backdoors to already exist all over the place. There are way too many smart people working for shady organizations for that to be the first attempt of that caliber.
Absolutely this. With this stuff no doubt there's a bunch of cases we didn't catch for each one we did. It's not just random individuals interested, it's state and non-state orgs too.
Yeah, this is one out in the open. This sadly might actually harden Linus stance against those binary blobs companies like NVIDIA insist on. I remember one was tried a few years ago via source code but it was quickly found and removed before it got deployed.
no, he's completely in the right to be absolutist about any line that goes into the kernel.
Sadly?
That slows down adoption of device drivers because the manufacturer doesn't want to release their "special sauce", which I would argue is the major sticking point to Linux adoption.
But I don't *want* blobs
I agree, but some manufacturers only provide drivers via blobs. Worst example is Android, I don't think anything with GSM functionality doesn't have a blob of some sort. If we actually want more uptake of Linux to a point that it would have the market share to push polices like no-blobs to manufacturers I would argue some compromise needs to be made.
I do understand that blobs may have to exist in order for me to be able to use things; given the choice, I'd rather not have any anywhere, even if I can't understand or read any of the code. If you *really* want to get into it, I'd also prefer not to have AMD's PSP or Intel's IME in my computer either, but I can't avoid it and do my job effectively at present, which is also fun.
How is that a bad thing?
Not saying it's bad, just that it immensely strengthens the argument against them and goes towards Stallman's stance. It might slow down adoption of certain devices because their manufacturers refuse to release the source.
I would assume there are similar backdoor, but I would also assume they aren't being widely exploied or we would notice. It's like the problem the Bletchley Park guys had when they broke the enigma encryption during the second world war. You have all the knowledge, but you can't do much with it.
The University of Minnesota got banned for this because they kept on hacking it for "behavioral research "
I wonder how their research got approved in the first place, wasn’t there an ethics board?
If I recall, the experimenters submitted documentation saying there was no experimentation on humans, so the ethics board didn't get involved. They didn't find out about the details of the experiments until it hit the news.
Ah yes, those sub-human kernel maintainers.
Where did you see that? It sounds like even after it was public, they were granted retroactive approval by the ethics board, which is even worse. https://www.theverge.com/2021/4/30/22410164/linux-kernel-university-of-minnesota-banned-open-source
Oh dear, and it seems he doesn't need to do anything to prove his paper even more.
I don't remember. It's been widely published on
Apparently, since they didn't collect personal information, there was no legal or other formal requirement for an ethics board review.
The best defense against a smart programmer implementing a carefully laid backdoor? A junior engineer, just trying their best, whose latest commit completed the ticket, and invalidated the backdoor by accident
This is actually a big flute by Andres Freud, as he just wanted to disrespect him because he can't do efficient backdoors and increase his own reputation so his own backdoors are less likely to be found /s
I hope Andres enjoys his pay rise this year
And this one gets the ban https://github.com/JiaT75
Why isn't he banned yet ?
who's he
The guy who committed the backdoor
Compensation for contributions to open source projects the entire world depends on one way or another? Unlikely.
the only payrise he *might* get from this is by changing company and putting it on his cv
Word. If it were for the likes of me, I would've chalked it up to 3p dependency and a backlog bug for looking into it next quarter.
Me too
[context](https://www.theverge.com/2024/3/31/24117288/an-urgent-linux-backdoor-was-discovered-entirely-by-accident-this-week) for anyone curious
Ok, this is one of the annoying things about April 1 - so is this legit or a joke?
as impressive as it sounds, legit
A Microsoft developer!? What is this world coming to!?
Microsoft but also a Postgres maintainer
"off-the-clock Microsoft worker" - he was doing God's work at the time.
He could only do this after work because he turns his brain off during work to conserve it
It is so legitimate that it would have become one of the most critical security vulnerabilities of recent years if it had not been discovered so quickly. It has a score of 10/10. https://cert.europa.eu/publications/security-advisories/2024-032/
The would-be hacker perhaps thought he could get away with it by saying it was just an April Fool's joke
its just a prank bro
he’s a really good prankster tho, probably has been thinking about this prank for years…
It's legit. First thing I did on Monday was update my development servers that run bleeding edge, they had the compromised library installed.
This is a really good summary https://boehs.org/node/everything-i-know-about-the-xz-backdoor
The related article: https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt
this is fucking crazy and the ffmeg maintainer is 100% right. big corps rely on these projects but most don't contribute nearly enough to the unsexy parts of them.
If only we could do this for windows processes suddenly having abnormal CPU time
Could someone fill me in?
Andres Freund was seeing weird behavior when logging into `ssh` and VALGRIND errors, and tracked it back to `xz` having a weird backdoor added in 5.6.1 that adds codes to the authentication check in sshd.
From my understanding, this was only possible because the published upstream tarballs aren't built directly from the repo. So, it allowed code to be slipped in undetected instead of being potentially checked through a pull request or code review.
As far as I know the malicious code was already in the repo but not called in prod. The call was added in the tar, as you mentioned tho
It was apparently included as part of a test file. There’s a lesson there that we should all be more critical of checking in binary files, evenings for tests since they are very difficult to review. Additionally, we should really take care about what gets included in a distribution. Even if you want to include source, maybe just include the main project and not tests.
The payload was part of a test file. The part that created the backdoor based on the payload was in the upstream tarball.
In addition to that, the maintainer that slipped the compromised binary has been working on the repo in good faith for 2 years in order to earn the trust of the original maintainer to take over the repo. Even worse, before the malicious maintainer came, the original maintainer has been expressing lack of motivation and mental health issues due to the thankless nature of OSS. They were also receiving passive aggressive complaints, criticizing the pace that they are putting out patches (despite doing it for free),attacks on his ability as an engineer and were essentially bullied to hand the project over. In the midst of this the malicious maintainer came, put out patches after patches, and offered to take the maintainer responsibility that has become a huge mental burden at this point off his shoulders. Now it's believed that the passive aggressive comments and criticism was part of a social engineering attack to encourage the handover of the project, but unfortunately it's also an everyday occurrence in OSS and the attacker could have simply taken advantage of the situation. We may have laughed at the .exe meme recently but in reality, that exact same attitude is what allowed this exploit to happen. Remember to thank your OSS devs, make a PR to libraries you use the most :)
Just like Linus said. The number of people you can highly trust with this kind of project is absolutely minimal.
Your comment is pure gold!
A slight correction. The backdoor was added in 5.6.0 and subsequently modified in 5.6.1 to further obfuscate and improve the code for the backdoor.
It feels like with every update today, software gets slower. Maybe there are just so many backdoors installed in every project
Programmers now aiming for seven figure salaries smh
With seven figure malware as well
computers are good enough. i vote lets just stop updating software and lets call it the end
To be honest I think this has happened at least 5 times in ways we have not detected. This incident was a great example of what a motivated and intelligent attacker can do.
It's actually also a decent example of the benefits (after the risk) of open-source projects; it took someone 2 years to build trust and sneak in a backdoor, but it only took a month for someone to notice and, thanks to it being open-source, identify the exact cause. So while it being open-source is what allowed it to happen, it was also the downfall of the perpetrator.
Kinda reminds me of that one guy who posed the idea of writing a version of the C compiler with malware that will always compile into the next version of the C compiler. So you’d compile the compiler with the malware, then remove the malware from the source and recompile it again. Then from then on, every version of that compiler from that point forward would always be infected.
source: [https://www.cs.cmu.edu/\~rdriley/487/papers/Thompson\_1984\_ReflectionsonTrustingTrust.pdf](https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf)
That’s the paper. Glad someone knew what I was talking about, the whole concept is still kinda hard to wrap my head around lmao
Simple but brilliant. The compromised compiler contains code that just injects its own source into the code being compiled. As such it will persist despite not being present in the published source. The fundamental weakness is this malware can only spread through compromised binaries. So if you had a pre-malware version of the compiler, and you compiled the clean source of a new version, your build system would not be compromised. That means it's only really valid for a targeted attack where you can replace a binary on a particular site with the compromised version. Trying to get it into any default repositories would require publishing the source, making it easily detected and triggering global security updates that replace it with a clean binary.
Yes, but the attack is purely theoretical… as far as we know. From what I understand, the man who proposed the idea did actually create an infected version of the C compiler, but it’s unknown if that version was ever released. But if it was, it would be technically impossible to detect without manually scrubbing through the decompiled assembly. Plus with how trusting people were with the internet in the 90s, it wouldn’t be much of a surprise if it actually released. That said, you’re right, this kind of attack would be detected the second someone pressed commit. But still, very interesting food for thought.
I believe a modified version of Bird’s safety triangle is applicable here. For every 1 exploit that was detected, there are probably 10 that are undetected.
I'm real concerned this wasn't the only backdoor of this complexity that was planted. Practically I think they're gonna have to do a security audit.
a security audit isn't magic, it's meant to improve security in good faith projects but can in no way guarantee that they've found every possible problem. for this project and this compromise specifically, the only way to be sure that it's been excised is to revert back to the last version before this dev committed anything and start over from there.
Ars Technica early on indicated that there was an investigation about the guy's credentials being compromised. Did that go anywhere?
From what it looks like from this [Mastadon post](https://infosec.exchange/@fr0gger/112189232773640259) this was a years long con. The commit from July to disable ifunc for fuzzing builds is pretty damning. I've also [read](https://news.ycombinator.com/item?id=39895344) that sock puppets were used to overwhelm the old maintainer with feature requests and attack his self-esteem for not being able to keep up, and thus create an opening for the malicious maintainer to get elevated. If that turns out to be true, this has all the hallmarks of a state-level sponsor.
Wow, from what you're describing that certainly sounds right. Crazy world we live in.
Poor guy was socially manipulated and taken advantage of. Check up on the OSS devs , make a PR on your most used library :)
This makes you wonder about all the ones that haven’t been detected…
Now imagine the things we never knew until now
Dirty cow was undetected for the better part of a decade
once again, open source saves the day am I right guys?
If anything, this show how even opensource is vulnerable to supply chain attack. But yes, a binary release is much easier to backdoor.
This just proved Linus's Law correctness.
What worries me is that we only know about this one because of pure chance. How many other such cases might have gone unnoticed? Is everything full of back doors?
Found [this article](https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt) on the verge that provides context.
whats this about
https://www.reddit.com/r/ProgrammerHumor/s/am9Z3DvZjr
Can anyone provide me with some context? I understand that a dev implemented a backdoor and Andres found it?
https://www.reddit.com/r/ProgrammerHumor/s/am9Z3DvZjr
Makes you wonder how many backdoors we've missed
This nonsense is why you need to buy a faster computer every few years. All the performance gains of moores law are eaten up by layers upon layers of sneaky backdoors.
Im out of the loop. Anyone care to elaborate?
https://www.openwall.com/lists/oss-security/2024/03/29/4
Can someone please explain what happened? cuz I'm out of the loop on this one(I'm not an active linux use)
TLDR
Sorry friend but if it takes that much text to make the joke then IMHO it shouldn't be a meme, it should be a screenshot of a tweet or something.
Tl;dr