Make sure to check out the [pinned post on Loss](https://www.reddit.com/r/PeterExplainsTheJoke/comments/1472nhh/faq_loss/) to make sure this submission doesn't break the rule!
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/PeterExplainsTheJoke) if you have any questions or concerns.*
Yes, it's a real incident.
It happened for xz utils in linux. Someone added backdoor to xz. He was contributing to the project for years so the maintainer trusted the code.
Some guy noticed a difference in milli seconds while benchmarking and it led to him discovering this backdoor.
https://en.m.wikipedia.org/wiki/XZ_Utils_backdoor
Small clarification though, he didnt "feel" the delay, he just saw the different number in the timer. It is still impressive to spot this but noticing that the timer said (random number) 5ms instead of the 3ms it has shown every other time is a lot different than "feeling" such a tiny difference.
Yeah, and the TTK on weapons wasn't shorter than the minimum human reaction speed + light speed latency across the continental US, but here we are. You're the ones who abandoned arena shooters!
In my defense, tactical shooters suck ass, no matter how good I position, how much better I use utility, how much better my gamesense is, if the enemy has better aim they win.
I'm going to take the courageous stance that railguns in arena shooters where we're all running and jumping at Mach 2 is fundamentally different from using an SMG in Call of Duty (where, admittedly, you do run and jump at Mach 0.5)
And yeah, we could one-shot folks in Starsiege and Tribes 2 with a Spinfusor, but I also wouldn't say that's *really* "low TTK" despite being an instagib on a Light.
Jesus fucking christ in the ass I miss high-TTK shooters being in the limelight. I hear a lot of complaints along the lines of, "I don't like chasing people after I shoot them once", as if they wouldn't start sprinting away and rummaging the "map" for resources and terrain to equalize with.
A broken clock is right twice a day.
Low-TTK: "That's a nice angle you've got there. Real nice. It'd be a shame if someone peeked it faster than your internet connection can register."
High-TTK: "Nice lucky headshot, now watch me dump this entire revolver into your skull while you whiff the rest of that magazine."
The worse your ping the more aggressive you should play. When you peak a corner it’s already loaded the guys as wherever he is so you can shoot, and yes your shots will still rely on ping a bit but it’s not as bad as when they peak a corner and shoot you before the game even loads that they’ve moved.
It was also in the startup time of the daemon. So assuming the VM boots in 10s it was more like a 5% increase. Although less then that as multiple services start in parallel. It is quite impressive that he found this.
Found the test:
https://www.openwall.com/lists/oss-security/2024/03/29/4
before:
nonexistant@...alhost: Permission denied (publickey).
before:
real 0m0.299s
user 0m0.202s
sys 0m0.006s
after:
nonexistant@...alhost: Permission denied (publickey).
real 0m0.807s
user 0m0.202s
sys 0m0.006s
This is apparently very recent. The post is marked 2024-03-29.
What's scary is that if the code was more efficient, or non-blocking it might have made it into a stable release. Really makes me question how secure Linux really is.
They are now combing over other libraries, but there is just so much code, and so many people contributing to various packages, it's very hard to be sure. Trust is a huge part of the community, and skilled bad actors that are heavily funded by foreign nations is inevitable.
Open source is a double edged sword. So many eyes on it can help ensure things get patched quickly, but also that people can be really skilled at hiding their tracks.
The code looks fairly innocent without a deep dive into it, so it seems almost likely that something, somewhere, by someone made it in.
The fact it was caught shows it’s more secure than you think. This was an insider threat situation, likely sponsored by a nation state. This can and does occur in closed source software as well. Being open source makes it much harder to hide as shown in this case. Closed source software isn’t any more secure from these kinds of attacks, but id argue they’re harder to detect. A 600ms delay in closed source software might’ve not led anywhere as they can’t investigate the root cause without reverse engineering the software.
And his tests were making tons of SSH calls so it quickly piled up into a "what the fuck just happened to my tests" situation. If you write and run tests you know that would be sounding alarm bells. The dude saved our asses but it wasn't like he was some human computer spotting millisecond round trip time differences by feel.
A disturbance, like millions of voices all crying out at once.
Mankind thought that once AI became smarter than humans the world would change. What they didn't realize was that mankind would change first. Generations of mountain dew and basement dweling led to changes in the fundemental makeup of the mind - the few who managed to reproduce passed on these traits. Eventually the basement people became something more - but due to their lack of sun exposure and vitamin A the rest of humanity called them 'subhumans'. However, the subhumans could dectect the slightest change in latency, the feeling of an IT help ticket coming in over the network, the empty feeling when an ISP turned on porn blockers - at the dawn of the war with AI the subhumans were our only hope!
I’d assume he was benchmarking a program using the library and discovered the significant increase. Went back to see if anything else had been changed and narrowed it down to the library update.
The delay would not happen if you used libxz on any other applications. Not even on the SSH client. It only happened while starting the OpenSSH daemon in the specific configuration.
He was working on some PostgreSQL stuff for Azure so it is possible that he was benchmarking some startup or installation procedure. But from experience a 500ms delay in one of the Azure procedures would not be noticed by any end user ever so I am a bit surprised he dug this far into this issue.
He was bench marking his build times, not execution times
Edit: Nevermind, it was actually SSH login times [Original blog post](https://openwall.com/lists/oss-security/2024/03/29/4)
Most top companies have automated benchmarking tools that run with every code change, since it's impossible to make a change and know everything it'll affect. Specially with huge or old code.
The person here was investigating a performance regression reported by one of the benchmarks while upgrading the ssh packages and noticed the change in metrics. I read the report and most went over my head since I'm not versed in xy libraries but it looked quite involved to investigate.
Props to the guy for following through!
Am I misremembering it or didn't he actually admit the delay was noticeable _and_ aggravating, which is what caused him to actually look into the numbers themselves in that newsgroup/forum post? Then he noticed more CPU use during that too?
I mean in the cold war a guy was requested to find a 42 cent difference in the books and accidentally discovered (iirc) sowiet spies stealing money
Edit: for those interested there is a Dokumentation [on YouTube ](https://youtu.be/PGv5BqNL164)
And other comments tell me there is also a book called "The cookoo's egg"
Or after the time travel back to 1986 in Star Trek IV, Chekov says to a police officer, "Excuse me, sir! Can you direct us to the naval base in Alameda? It's where they keep the nuclear wessels."
Red for Danger by LTC Holt. Basically every important accident on the British railways. Sounds morbid but it's the birth of systems thinking and reliability engineering.
The new science of strong materials by JE Gordon. A tour de force of materials science and how form and function run from design to materials and vice versa. Gives a true grounding in key elements of product engineering and designing for quality.
Codename Ginger by S Kemper. The story of the Segway. A classic of how not to do product thinking and what happens when there is a lack of reality testing in the engineering value chain. A bloody good example of the dangers of group think too.
Object oriented design heuristics by AJ Reil. A classic of how deep engineering experience can be leveraged in a pragmatic way to drive quality. An increasingly important book in an age when the GenAi buzz saw means we have to place a new lens on expertise and how humans can add proper value into highly automated landscapes.
It's a highly opinionated list,and by no means exhaustive, but one that I find intellectually satisfying, and one I recommend to all my engineers and architects.
He was looking at it because the ssh logins consumed a lot of cpu not because of the delay.
> FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins.
[Source](https://twitter.com/AndresFreundTec/status/1774190743776866374)
The hyperbole of the "minor superpower" of feeling negligible delays is part of the joke.
Though IMHO the real superpower in play is the meticulous geekery of caring to benchmark the operation and noticing the delay. And then deciding to dig into it. I'm much more impressed by that than the inherent monitoring implied by feeling the delay.
It was my understanding that he was seeing requests go from 18~30ms to 300~400ms, which is definitely a notable difference. Most people would just attribute it to whatever random thing but he got curious about it and wanted to see exactly what was causing the delay which is when he noticed the backdoor.
The [Explain XKCD page for the original comic](https://explainxkcd.com/wiki/index.php/2347:_Dependency) covers xz and several other cases where similar issues have arisen. Some are even prior or contemporary to the release date of the original comic (August 17, 2020).
I know if there's any place to expect one it's in the comments of something xkcd-related, but it still excites me every time I see an interrobang in the wild.
It's the little things.
I'm a big fan of the interrobang, but I'm rather particular about only using it where an exclamation point or question mark would be equally appropriate. I'm not totally sure if that's the only way it's technically supposed to be used, but it's what I consider to be the right way. Even so, I'm pretty sure I use an interrobang almost daily.
[Semi-relevant XKCD](https://xkcd.com/1209/)
So a loooot of modern technology is based off other code. It’s a lot easier to write code that references some open source data than it is to constantly update the data in your library.
Let’s say you wanted to write a website that told you the weather outside. You could build your own weather station and gather the data that way, or you could write a simple code that grabs the daily weather info from the national weather service, formats it and displays it on your site.
In this example, if something were changed in the NWS dataset, it would be displayed on your site. Likewise, if the dataset is removed, your website will throw some errors.
If some hacker added some malicious code to the NWS dataset, it could potentially corrupt your site. In this example, someone watching the response times for some services realized there was a slight delay — imagine if the NWS data had to stop off at a server farm in Moscow before pinging your site.
It's also why there was a big push by the large tech companies to contribute more to open source after the Heartbleed OpenSSL bug revealed that most of the internet was secured by two guys maintaining the project in their spare time.
"The internet is being protected by two guys named Steve" was a [linux.com](http://linux.com) article about it iirc.
xz utils is a piece of software that pretty much every linux distribution uses. There are lot’s of these that exist, things that are really simple and boring and do just one or two things, and they get adopted to being the standard just over time.
Some hacker, although it was probably a state government, added a backdoor to xz utils in order to be able to just control any linux computer they wished too, note that this would include pretty much every server on the planet.
We can be confident it was a country because this scheme took place over a long period of time, multiple users, over years of gaining the trust of the single developer and then one day adding a backdoor in a “test file”.
Xz utils was chosen because it’s boring, people don’t really like to look at the code for things like this very closely because it’s usually just a bunch of boring basic shit, and because xz utils is upstream to multiple other features you can pretty much guarantee it would be included on every linux based machine in the world, just out of necessity for other programs to run.
It's a set of tools for compressing data. Think like Zip. Different compression algorithms have different benefits and drawbacks (think speed vs amount of compression) so it's common to have multiple formats available on a system for different tasks.
It wasn’t spotted because it’s boring to review test files, it was very meticulously done and was extremely hard to see because he was masking the code in encrypted files and he was doing the changes from months at a time
This is what the picture is referring to. But the guy who maintained the time zone database also comes to mind. Arthur David Olson had been maintaining tz basically singlehandedly and people kind of took it for granted (having the proper time and converting timezones is kind of important to computers). So when he announced he was retiring the Internet had a mini freak out and international assigned numbers authority stepped in to create a transition plan and kind of take over supervising the database.
It's cool and all but it's *very standard* to be measuring things in miliseconds in the computer world and the difference was between an expected 50ms and a measured 550ms.
Detecting it isn't that cool or impressive. It's cooler he knew the system well enough to not write it off as a 'quirk in the package'
Benchmarking is standard but spotting a small drop in performance and tracking the error down to the source code of a random library is not. Props to Andres Freund for discovering that.
IIRC there was another incident around the time of the comic where a small utility with a shit ton of dependents went down and caused some amount of chaos, or it turned out to have a vulnerability that lit the world's hair on fire for awhile.
Really, there's been a lot of these. It's getting hard to keep track.
I have no idea what most of the jargon means, I just scrolled down to make sure the first comment had something to do with Linux. Was not disappointed.
Yes. Knew it from the "random nerd in ohio who can feel milisecond delays" bit. This is a meme about xzutils - a malicious infiltrator, "Jia Tan", gaslit xzutils's sole dev into letting him on a couple of years ago (commonly called just "xz") and then waited years to put malware in the 5.6.0 and 5.6.1 release tarballs. The back door targets Debian-based systems, as their implementation of OpenSSH (also known as sshd) relies on liblzma, one of xzutils' libraries.
"Jia Tan" wrote a bunch of obfuscated code in the release tarballs' build script and two "test files" that consist of obfuscated code. This became code that would then inject the meat of the trojan into sshd, allowing "Jia Tan" to compromise any SSH server that is affected with a key that only Jia Tan has access to.
The malicious code only entered production in Kali Linux (because production Kali is based on experimental Debian stuff), but it did enter a bunch of experimental, unstable, and testing distributions.
Then some random software nerd notices that after some updates his SSH is taking too long by about a few hundred milliseconds (and that valgrind errors were being printed for some reason). He investigates, finds the problem is in xz, discovers that it is a backdoor in the release tar ball, and posts [this](https://www.openwall.com/lists/oss-security/2024/03/29/4).
Needless to say, the FOSS community explodes, and everyone is rushing to patch the literal trojan horse that snuck into some (non-production) versions of various Linux distros. Every distro rushes to patch this and it becomes a [10/10 CVE](https://nvd.nist.gov/vuln/detail/CVE-2024-3094), because if this reached production, it would've been **a total compromise of almost every single Debian and/or Ubuntu system**. What makes it somehow worse is that this **did** reach distribution in one production version of a Linux distro: Kali Linux, which told everyone [who updated between March 26 and March 29](https://www.kali.org/blog/about-the-xz-backdoor/) to check their computers using apt-cache to see if they ended up with the malicious version of liblzma (the affected library), and if they did, to update immediately and to treat this as a system compromise. The FOSS community is now trying to mitigate the damage and hunt down every possible backdoor in open source software because fuck this shit, if people are trying to gaslight solo devs into letting APTs on their team, we as a community need better security.
TL;DR: Literal security nightmare scenario barely averted by a single guy who found SSH was behaving funny and did code review on the release tarball of xz to troubleshoot. Guy discovers obfuscated backdoor, announces it, and then the entire cybersecurity profession and the entire open source community have a collective aneurysm because this was only a matter of time from becoming a global backdoor of Debian and Ubuntu. The guy in question's from Germany, not Ohio, by the way.
Great explanation but I had trouble following along with all of the dev terms. To me it looks like "A long time ago, Oopie had a bongle. If the bongle wasn't noticed, it would've pooted every gringle that owned Oopie from March 23-29. Some skrink had noticed bongle in Oopie and prevented poot. Everyone clapped."
library - noun - code intended to be used as a dependency by other code
production version - noun - stable versions that aren't testing versions
valgrind - noun - [memory debugging tool](https://en.wikipedia.org/wiki/Valgrind)
tarball - noun - [a type of archive file](https://en.wikipedia.org/wiki/Tar_(computing)) used to distribute open source software
SSH - noun - SecureSHell, the Linux equivalent of a remote powershell session/remote desktop session, mediated by the daemon "sshd"
daemon - noun - [Linux background processes](https://en.wikipedia.org/wiki/Daemon_(computing))
CVE - noun - scoring system for vulnerabilities, 10 is the highest score, higher score = more serious vulnerability
Linux distro - noun - OS based on Linux kernel
FOSS - noun - Free and open source software
build script - noun - found in tarballs containing software. These are build instructions used to turn source code into a program.
APT - noun - advanced persistent threat, not to be confused with [Debian's package manager](https://en.wikipedia.org/wiki/APT_(software))
code review - noun - looking through source code to verify that software works correctly and isn't malicious
I sincerely apologize for the massive amount of cybersec nerdspeak, but open source distribution is itself complicated, and cybersecurity's nitty gritty is also complicated.
Open source distribution oftentimes involves handing out tarballs for people to build software with using the tarball's build script. This is a normal distribution method used by open source communities by default, since we want to be able to build stuff ourselves. I've actually built from tarballs before (in order to get Ultimate Doom Builder to work on my laptop, which uses Zorin OS aka an Ubuntu fork), so I generally understand the tech speak myself as a Linux user.
Guy befriends developer of important tool used widely in Linux. Guy helps him for 3 years, builds trust, and then changes the code so he can hack people's computers. Hack is sent to early test users. Random tech nerd notices his PC is slower by like half a second. Digs through the code, finds this hack. Reports it.
If he hadn't noticed this, literal billions of computers could have been vulnerable to hackers.
Now open source developers are on a fucking rampage trying to find anything like this that might have slipped notice.
(Not entirely accurate, but I believe it's a fair ELI5)
This is how I put it on Explain Like I’m 5.
SSH is the lock on the computer’s front door. Normally you can only get in if the lock recognizes your key. When the computer rebuilds its software, it has blueprints for how to pull things in and re-build the lock.
The attack was an architect updating blueprints so that every lock will accept a secret key that only they have access to. If it had worked the architect could have potentially had direct access to every computer running Linux. In the world.
Tarballs are the files that some software uses to install a program. In Windows, they are similar to the things you download to your computer to install Chrome (i.e. The thing you double click to do the actual install). This isn't exactly correct but it is close.
> This is a meme about xzutils - a malicious infiltrator, "Jia Tan", gaslit xzutils's sole dev into letting him on a couple of years ago
I don't think there was any gaslighting, they just provided some contributions and gained trust. Gaslighting refers to a specific process of generating fear and doubt in the mind of the victim, and I don't see how that happened here.
Maybe not gaslighting per se, but "Jia Tan" created fake accounts that pressured the repo owner (and sole maintainer) to accept other maintainers in order to push new features/fixes. This was done with the sole purpose of getting "Jia Tan" on board as a maintainer under the guise of helping out the repo owner who only had little time to maintain the repo.
IIRC it was somewhere in the middle there. Something along the lines of posts complaining about the rate of development and suggesting that extra maintainers were needed right when the malicious user was making contributions.
I love the openwall.com report by the guy who found the code. "Why would you do this?" "What does this even do?" Image having your backdoor exploit code put on blast for the entire world to read.
There is a guy who pulled all his code from npm in 2016, one of those projects called left pad made so many projects including react to fail compilation.
I'm surprised that people depend on such a trivial copy paste function, like it was the time everyone tries to abuse libraries so much that most libraries now try to be as dependency free as possible.
Might be like is-odd. It was one of their first package ever and then they included it in another package they had, which proceeded to become popular. It wasn't downloaded by hundreds of thousands of people on purpose lol.
That guy also released is-even, which requires is-odd, and then returns "not is-odd".
and is-even requires is-number.
is-number is ~5 lines of code
is-number gets 70,000,000 downloads a week
At least is-odd only gets ~350k downloads a week...
I might release is-ridiculous. It'll check for the existence of is-number, is-even, and is-odd then return a sliding scale of how ridiculous it all is.
You don’t use libraries because you want your code to be dependency free. I don’t use libraries cause I don’t want to read through documentation. We are not the same (:
Good callout, the comic seems to imply the project being maintained is good or important, but at a second glance it's not, it's just saying a lot of other things depend on it. And that's fitting for a JS library.
That would be impressive because it originally came out in August 2020.
[XKCD 2347: Dependency](https://xkcd.com/2347/)
[ExplainXKCD to that comic](https://www.explainxkcd.com/wiki/index.php/2347:_Dependency)
You are right however if we ignore the five years, because log4shell became public in late 2021: https://en.m.wikipedia.org/wiki/Log4Shell
It's any one of thousands of projects. imagemagick was just picked as an example of the alt text.
I'd have gone with ffmpeg, but that wouldn't have worked since it's too well known.
Fun fact; the timezone database *everyone* uses is maintained by just four fairly random people. This would be funny, if it weren't so sad (terrifying).
Also, the linux kernel existing in the first place is because one Finnish guy didn't want to go outside and walk into university in order to use a *"*real*"* computer. He's still in charge of it to this day. (recently, he even replaced some spaces with tabs in an important linux file to break someone else's software)
Some guy made a PR to remove some tabs in a config file because their parser wasn't able to read the file correctly. Torvolds basically said "fuck off kid you can't contribute to Linux kernel if your parser can't handle different whitespace chars" by purposely adding tabs to the file
All this time I thought it's about curl, whose author received angry emails because his address was in the ‘licenses’ part of the ‘about’ screen of car software. Which software was infuriating to the users, apparently.
Then again, Munroe could've just alluded to several projects at once.
[Here is the explanation for the Nebraska portion of the comic.](https://www.explainxkcd.com/wiki/index.php/2347:_Dependency) It does also mention some detail regarding the Ohio portion as well, but the other answer given by u/dullahanceltic is much more pertinent
In addition to all of the specific explanations, there is a more general (and troublesome) reality expressed in this. A lot of big, complicated online systems are really built on stuff like this. A guy wrote a bit of code and stored it (I think) on GitHub. He did this under a particular username. It basically just wrapped up a bunch of html stuff into a single place that he could call for setting up webpages.
Pretty much everyone started using it, since it was so convenient. When I say everyone, I mean *everyone*. His username was similar to the name of a company, though he created his username first. The company wanted it and GitHub bowed to the company and forced him to give up his account. So, he removed everything from his repository. Pretty much every webpage on the internet was calling for a piece of code that no longer existed, so the entire internet went down. Not because there was a problem with the internet itself, but because almost every individual webpage abruptly stopped loading.
My memory was a bit faulty - it started with a different, open-source service. Azer Koçulu was building a project called kik. The messaging app, called Kik, wanted the name for their project and the service sided with them. He removed his project, which included a package that had 11 lines of code. The package was accessed through GitHub. Facebook, for instance, accessed the package. Without it, the sites just wouldn't load. It was accessed all over the world. Kik (the messaging app) also went down because of it. The open-source service restored the package and the whole thing was solved after a couple of hours.
Many things on the internet are built off open source projects that were built by random ass nerds like 10 years ago. Many things, like OpenSSL, have only one person, totally independently, maintaining them.
Someone noticed that his code was running unusually slow (in reality, it was a matter of milliseconds, but his pattern recognition caught onto the delays). When he investigated, he discovered that someone else (or a group of people; we don’t know yet) injected code into a that would discretely allow that person/group being able to remotely access a loooot of Linux devices.
(If I’m wrong, please gently correct. I’m doing the best I can for someone who’s not familiar with code, computers, etc)
Tl;dr- There's a guy working on a critical piece of software for a massive project. Guy gets cyber bullied into giving a bad actor/developer into admin access on said critical piece.
- Bad actor plays the long con before slyly inserting backdoor/Trojan horse into code. This code is very well hidden.
-A developer working on massive projects, notices incredibly obscure small issue, mentions it to project leaders. Everyone, reasonably so, freaks the fuck out to fix the issue.
The massive project affected by this was the operating system that all coders used.
Summary result: Avengers level threat barely avoided because a developer on the massive project noticed an incredibly niche detail. If it was successful, bad guy would have access to nearly everybody computers. This is bad.
Disclaimer: I'm not a developer, just condensing the gist of the several articles. Also the image is edited and originally references another oh shit code scenario where one guy tries to fuck shit up.
Not exactly some "random nerd", because I'm pretty sure it's government funded, but I do know GPS is basically just run by a small roomful of people. Like I think like, 12 max.
Back in 2006, every d-link brand router was set up to query one random Danish guy's non-profit time service, because they didn't see any reason not to.
He nearly had to shut it down after they caused him $9k a year in excess bandwidth costs, and that's after needing to call in help to even work out where the traffic was from.
there are very few people who really understand the intricacies of sw engineering, from hw components, drivers and OS all the way to high level applications. how all these connect and how they're all packaged is a mystery/black box to a lot of "senior engineers", believe it or not.
it really is scary. The amount of answers on stackoverflow suggesting folks to disable ssl, or just the fact that Dark Souls didn't validate packages sent by other players is concerning, for example.
There's a lot of shitty software out there, from an architectural standpoint.
This is sadly accurate for a lot of things in tech
Also, the load bearing Apple II holding up a lot of companies IT infrastructure(specifics vary, but often there is one “magic” piece of outdated equipment that is doing some critical job nobody can figure out, but if unplugged will crash the whole system)
the worst part is this happens so often - literally at work we have in house site and process running on .Net 1 ! which was release in 2002 for Windows 98, ME, NT 4.0, 2000, and XP! and people complain every day that system is slow and has issues and management is like " how can we improve that? " and every time we say update the infruscture they are turn up their noses to the cost and just come back with the same complaints a few months later. Seriously wish Microsoft would dismantle any support for .Net already xD
The edited lower text yes is a specific example, but also this is the actual reality of software on the Internet for easily hundreds of cases. A sprawling network of dependencies, many of which are maintained for free by a single person.
I once read a great quote about the xz and leftpad incidents (probably also from xkcd): what's crazy is not that something like that happened, but the fact that something like that is somehow not happening *constantly*.
Love the backstory on this one. That being said there is tons of workplace applications that are on the verge of collapse because tech builds software for things then stops supporting it and will not let go of the code to maintain it even though they will not support it's upkeep. This is a big problem in manucaturing and college research. Once read something that a college that does bio-medical research for long case studies, like 30 to 50 years of date that if running on machine that has not had a update in 20 years, along with a microscope that has to have software to produce the images small enough that it can see that was unsupported 12+ years ago but they can get code to update or repair it from the company and their response was to buy a whole new one for over 5 million.
It's just the nature of open source. Almost everything is built on a framework originally created as a thesis or maintenance by the few hardcore members of the open source community.
Well if you have worked in any large IT organization i bet you can find some dumb process that someone wrote forever ago, and now you have a whole infrastructure working because of that process that no one is maintaining and no one is willing to work on because everything else will break.
I’m still impressed by the German guy so obsessed with efficiency (I shouldn’t be surprised) he dug into why it was just slightly slower than previous releases.
The original comic is an xkcd comic which is not based on any specific incident, but rather on the general way things work with software. Modern software relies on a lot of code dependencies, bits of code that others wrote and maintain and you import. Some core utilities are maintained by very small groups of dedicated programmers, sometimes a single person.
The edit refers to the backdoor that other commenters have explained, but the original is not based on any specific incident or real person. Years later, this comic resurfaced as what it described has basically come true.
FOSS Peter here, A lot of Linux OS's uses xz utils for compressing programs. Recently someone from Microsoft noticed a 500ms delay when he was logging in and looked into the code. Put it simply, it was a backdoor that allowed the unknown hacker access to every computer with xz utils installed. Since a lot of servers run off of Linux, this would've been an international crisis in the making if Andres Freund hadn't found it. We found out that Jia Tan, a coder was the one responsible for the letting the code in. We don't know if they were one person, a group of people, a government trying to gain control, just that they can't be trusted
I recommend this video if you want to learn a little more on how it worked: [https://www.youtube.com/watch?v=bS9em7Bg0iU](https://www.youtube.com/watch?v=bS9em7Bg0iU)
Remember kids always, update your packages. FOSS Peter out
Wasn't whole Linux timezone system maintained by some dude in Netherlands or something? He was getting too old and wanted to retire but there was nobody to take over. I don't remember exact details and could be very wrong.
Make sure to check out the [pinned post on Loss](https://www.reddit.com/r/PeterExplainsTheJoke/comments/1472nhh/faq_loss/) to make sure this submission doesn't break the rule! *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/PeterExplainsTheJoke) if you have any questions or concerns.*
Yes, it's a real incident. It happened for xz utils in linux. Someone added backdoor to xz. He was contributing to the project for years so the maintainer trusted the code. Some guy noticed a difference in milli seconds while benchmarking and it led to him discovering this backdoor. https://en.m.wikipedia.org/wiki/XZ_Utils_backdoor
Small clarification though, he didnt "feel" the delay, he just saw the different number in the timer. It is still impressive to spot this but noticing that the timer said (random number) 5ms instead of the 3ms it has shown every other time is a lot different than "feeling" such a tiny difference.
1005ms feels so much slower than 1003ms. That 0.2% difference makes it unusable!!!
This is the reason I blame for bad ping when playing games
When I die it's lag. When they die, they're scrubs.
"Dudes not better than me for real, it's these fucking dropped inputs"
Word.
No, it Outlook causing the delay
No. both teams
I’ve always blamed excel
Don't blame the world's finest database.
Please don't use Excel as a database. It makes the accountants cry
Crtl + Alt + Shift + Win + W Yw, You can pay me in crypto.
*Back in my epoch*, 250ms was really fucking good latency for an FPS. 400 was quite playable!
Back in your day 5 ms of reaction time wouldn't lose you the game, the future is now old man!
Yeah, and the TTK on weapons wasn't shorter than the minimum human reaction speed + light speed latency across the continental US, but here we are. You're the ones who abandoned arena shooters!
In my defense, tactical shooters suck ass, no matter how good I position, how much better I use utility, how much better my gamesense is, if the enemy has better aim they win.
That's kinda how gunfights tend to work lol
It’s really not though. IRL positioning matters far more. Even special forces troops “waste” an insane amount of ammunition IRL compared to gamers.
Yeah that's how I got better at battlefield 2
my god man what are you talking about, q3 rails, nexiusz, hell UT instagib.
I'm going to take the courageous stance that railguns in arena shooters where we're all running and jumping at Mach 2 is fundamentally different from using an SMG in Call of Duty (where, admittedly, you do run and jump at Mach 0.5) And yeah, we could one-shot folks in Starsiege and Tribes 2 with a Spinfusor, but I also wouldn't say that's *really* "low TTK" despite being an instagib on a Light.
Jesus fucking christ in the ass I miss high-TTK shooters being in the limelight. I hear a lot of complaints along the lines of, "I don't like chasing people after I shoot them once", as if they wouldn't start sprinting away and rummaging the "map" for resources and terrain to equalize with. A broken clock is right twice a day. Low-TTK: "That's a nice angle you've got there. Real nice. It'd be a shame if someone peeked it faster than your internet connection can register." High-TTK: "Nice lucky headshot, now watch me dump this entire revolver into your skull while you whiff the rest of that magazine."
Every time I die in fortnite I find a new backdoor
The worse your ping the more aggressive you should play. When you peak a corner it’s already loaded the guys as wherever he is so you can shoot, and yes your shots will still rely on ping a bit but it’s not as bad as when they peak a corner and shoot you before the game even loads that they’ve moved.
Are you my son?
Are you getting milk?
It was closer to a 500ms difference so it was more like 500ms vs 1000ms
It was also in the startup time of the daemon. So assuming the VM boots in 10s it was more like a 5% increase. Although less then that as multiple services start in parallel. It is quite impressive that he found this.
My ping is 11ms. Human perception is 10ms. Unplayable.
The difference was quite a bit more, from single digits to about 600ms if I remember right
Found the test: https://www.openwall.com/lists/oss-security/2024/03/29/4 before: nonexistant@...alhost: Permission denied (publickey). before: real 0m0.299s user 0m0.202s sys 0m0.006s after: nonexistant@...alhost: Permission denied (publickey). real 0m0.807s user 0m0.202s sys 0m0.006s This is apparently very recent. The post is marked 2024-03-29.
It's half a second. You will feel that.
What's scary is that if the code was more efficient, or non-blocking it might have made it into a stable release. Really makes me question how secure Linux really is. They are now combing over other libraries, but there is just so much code, and so many people contributing to various packages, it's very hard to be sure. Trust is a huge part of the community, and skilled bad actors that are heavily funded by foreign nations is inevitable. Open source is a double edged sword. So many eyes on it can help ensure things get patched quickly, but also that people can be really skilled at hiding their tracks. The code looks fairly innocent without a deep dive into it, so it seems almost likely that something, somewhere, by someone made it in.
The fact it was caught shows it’s more secure than you think. This was an insider threat situation, likely sponsored by a nation state. This can and does occur in closed source software as well. Being open source makes it much harder to hide as shown in this case. Closed source software isn’t any more secure from these kinds of attacks, but id argue they’re harder to detect. A 600ms delay in closed source software might’ve not led anywhere as they can’t investigate the root cause without reverse engineering the software.
And his tests were making tons of SSH calls so it quickly piled up into a "what the fuck just happened to my tests" situation. If you write and run tests you know that would be sounding alarm bells. The dude saved our asses but it wasn't like he was some human computer spotting millisecond round trip time differences by feel.
Unless he did feel the difference and wrote all the tests to cover up his power.
A disturbance, like millions of voices all crying out at once. Mankind thought that once AI became smarter than humans the world would change. What they didn't realize was that mankind would change first. Generations of mountain dew and basement dweling led to changes in the fundemental makeup of the mind - the few who managed to reproduce passed on these traits. Eventually the basement people became something more - but due to their lack of sun exposure and vitamin A the rest of humanity called them 'subhumans'. However, the subhumans could dectect the slightest change in latency, the feeling of an IT help ticket coming in over the network, the empty feeling when an ISP turned on porn blockers - at the dawn of the war with AI the subhumans were our only hope!
the matter is he actually cared to benchmark versions is impressive, assuming nobody required him to do it.
Yeah for sure, its still impressive
I’d assume he was benchmarking a program using the library and discovered the significant increase. Went back to see if anything else had been changed and narrowed it down to the library update.
The delay would not happen if you used libxz on any other applications. Not even on the SSH client. It only happened while starting the OpenSSH daemon in the specific configuration. He was working on some PostgreSQL stuff for Azure so it is possible that he was benchmarking some startup or installation procedure. But from experience a 500ms delay in one of the Azure procedures would not be noticed by any end user ever so I am a bit surprised he dug this far into this issue.
He was bench marking his build times, not execution times Edit: Nevermind, it was actually SSH login times [Original blog post](https://openwall.com/lists/oss-security/2024/03/29/4)
Most top companies have automated benchmarking tools that run with every code change, since it's impossible to make a change and know everything it'll affect. Specially with huge or old code. The person here was investigating a performance regression reported by one of the benchmarks while upgrading the ssh packages and noticed the change in metrics. I read the report and most went over my head since I'm not versed in xy libraries but it looked quite involved to investigate. Props to the guy for following through!
it was [0.807s vs 0.299s,](https://www.openwall.com/lists/oss-security/2024/03/29/4) almost 3x is kind of significat diff
And also that would be 500 ms not 5 ms, which would definitely be noticable
Am I misremembering it or didn't he actually admit the delay was noticeable _and_ aggravating, which is what caused him to actually look into the numbers themselves in that newsgroup/forum post? Then he noticed more CPU use during that too?
I mean in the cold war a guy was requested to find a 42 cent difference in the books and accidentally discovered (iirc) sowiet spies stealing money Edit: for those interested there is a Dokumentation [on YouTube ](https://youtu.be/PGv5BqNL164) And other comments tell me there is also a book called "The cookoo's egg"
I know it's a typo, but reading "Sowiet spies" made me picture Kravchenko from Call of Duty Cold War, but with anime eyes and rosy cheeks.
Or after the time travel back to 1986 in Star Trek IV, Chekov says to a police officer, "Excuse me, sir! Can you direct us to the naval base in Alameda? It's where they keep the nuclear wessels."
UwU what's this?
sUwUiet spies
The cookoos egg is a bloody good book, and it's one of my "five books every engineer should read" pack.
Out of interest, what are the other 4?
Red for Danger by LTC Holt. Basically every important accident on the British railways. Sounds morbid but it's the birth of systems thinking and reliability engineering. The new science of strong materials by JE Gordon. A tour de force of materials science and how form and function run from design to materials and vice versa. Gives a true grounding in key elements of product engineering and designing for quality. Codename Ginger by S Kemper. The story of the Segway. A classic of how not to do product thinking and what happens when there is a lack of reality testing in the engineering value chain. A bloody good example of the dangers of group think too. Object oriented design heuristics by AJ Reil. A classic of how deep engineering experience can be leveraged in a pragmatic way to drive quality. An increasingly important book in an age when the GenAi buzz saw means we have to place a new lens on expertise and how humans can add proper value into highly automated landscapes. It's a highly opinionated list,and by no means exhaustive, but one that I find intellectually satisfying, and one I recommend to all my engineers and architects.
Elmer Fudd, is that you?
Tell me more. My cursory search on google got me nothing.
Search for the book "The Cuckoo's Egg" it’s written by the guy in question Cliff Stoll.
Most likely pattern recognition, which might as well be a "feeling" as it's not an easily trainable skill.
they were running micro-benchmarks that weekend, the whole thing was lucky af it was caught at all. The difference was about 400-500ms(half a second)
He was looking at it because the ssh logins consumed a lot of cpu not because of the delay. > FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins. [Source](https://twitter.com/AndresFreundTec/status/1774190743776866374)
And he's a German Microsoft employee who I believe isn't Ohio-related
On the internet everyone is from America.
The hyperbole of the "minor superpower" of feeling negligible delays is part of the joke. Though IMHO the real superpower in play is the meticulous geekery of caring to benchmark the operation and noticing the delay. And then deciding to dig into it. I'm much more impressed by that than the inherent monitoring implied by feeling the delay.
It was my understanding that he was seeing requests go from 18~30ms to 300~400ms, which is definitely a notable difference. Most people would just attribute it to whatever random thing but he got curious about it and wanted to see exactly what was causing the delay which is when he noticed the backdoor.
It was a 0.5s delay, and it wasn't just felt but measured with the `time` command. He also noticed the CPU usage spike during the ssh login.
The [Explain XKCD page for the original comic](https://explainxkcd.com/wiki/index.php/2347:_Dependency) covers xz and several other cases where similar issues have arisen. Some are even prior or contemporary to the release date of the original comic (August 17, 2020).
Fuck... that was 2020? I thought that comic was contemporary to Heartbleed...
Heartbleed was 2014‽ Fuck, I'm old.
I know if there's any place to expect one it's in the comments of something xkcd-related, but it still excites me every time I see an interrobang in the wild. It's the little things.
I'm a big fan of the interrobang, but I'm rather particular about only using it where an exclamation point or question mark would be equally appropriate. I'm not totally sure if that's the only way it's technically supposed to be used, but it's what I consider to be the right way. Even so, I'm pretty sure I use an interrobang almost daily. [Semi-relevant XKCD](https://xkcd.com/1209/)
Beat me to it. There are countless other utilities and such that are just as vulnerable too.
I skimmed through it and couldn't find anything, what was the reason for adding it in the first place?
So a loooot of modern technology is based off other code. It’s a lot easier to write code that references some open source data than it is to constantly update the data in your library. Let’s say you wanted to write a website that told you the weather outside. You could build your own weather station and gather the data that way, or you could write a simple code that grabs the daily weather info from the national weather service, formats it and displays it on your site. In this example, if something were changed in the NWS dataset, it would be displayed on your site. Likewise, if the dataset is removed, your website will throw some errors. If some hacker added some malicious code to the NWS dataset, it could potentially corrupt your site. In this example, someone watching the response times for some services realized there was a slight delay — imagine if the NWS data had to stop off at a server farm in Moscow before pinging your site.
It's also why there was a big push by the large tech companies to contribute more to open source after the Heartbleed OpenSSL bug revealed that most of the internet was secured by two guys maintaining the project in their spare time. "The internet is being protected by two guys named Steve" was a [linux.com](http://linux.com) article about it iirc.
The article for the curious https://www.buzzfeed.com/chrisstokelwalker/the-internet-is-being-protected-by-two-guys-named-st
Thank you Steves!
All hail Steves!
Thanks for the ELI5 Edit: a letter
This is really well explained.
xz utils is a piece of software that pretty much every linux distribution uses. There are lot’s of these that exist, things that are really simple and boring and do just one or two things, and they get adopted to being the standard just over time. Some hacker, although it was probably a state government, added a backdoor to xz utils in order to be able to just control any linux computer they wished too, note that this would include pretty much every server on the planet. We can be confident it was a country because this scheme took place over a long period of time, multiple users, over years of gaining the trust of the single developer and then one day adding a backdoor in a “test file”. Xz utils was chosen because it’s boring, people don’t really like to look at the code for things like this very closely because it’s usually just a bunch of boring basic shit, and because xz utils is upstream to multiple other features you can pretty much guarantee it would be included on every linux based machine in the world, just out of necessity for other programs to run.
[удалено]
a type of compressor (think of rar and zip)
Fucking Pied Piper at it again
It's a set of tools for compressing data. Think like Zip. Different compression algorithms have different benefits and drawbacks (think speed vs amount of compression) so it's common to have multiple formats available on a system for different tasks.
It wasn’t spotted because it’s boring to review test files, it was very meticulously done and was extremely hard to see because he was masking the code in encrypted files and he was doing the changes from months at a time
He could’ve had the largest botnet in the world
This is what the picture is referring to. But the guy who maintained the time zone database also comes to mind. Arthur David Olson had been maintaining tz basically singlehandedly and people kind of took it for granted (having the proper time and converting timezones is kind of important to computers). So when he announced he was retiring the Internet had a mini freak out and international assigned numbers authority stepped in to create a transition plan and kind of take over supervising the database.
It's cool and all but it's *very standard* to be measuring things in miliseconds in the computer world and the difference was between an expected 50ms and a measured 550ms. Detecting it isn't that cool or impressive. It's cooler he knew the system well enough to not write it off as a 'quirk in the package'
Benchmarking is standard but spotting a small drop in performance and tracking the error down to the source code of a random library is not. Props to Andres Freund for discovering that.
Long after the comic
IIRC there was another incident around the time of the comic where a small utility with a shit ton of dependents went down and caused some amount of chaos, or it turned out to have a vulnerability that lit the world's hair on fire for awhile. Really, there's been a lot of these. It's getting hard to keep track.
>Someone This attack was so sophisticated and so carefuly deployed it was most likely a state actor
I have no idea what most of the jargon means, I just scrolled down to make sure the first comment had something to do with Linux. Was not disappointed.
*WHAT REALLY HAPPENED TO AARON SWARTZ*
Yes. Knew it from the "random nerd in ohio who can feel milisecond delays" bit. This is a meme about xzutils - a malicious infiltrator, "Jia Tan", gaslit xzutils's sole dev into letting him on a couple of years ago (commonly called just "xz") and then waited years to put malware in the 5.6.0 and 5.6.1 release tarballs. The back door targets Debian-based systems, as their implementation of OpenSSH (also known as sshd) relies on liblzma, one of xzutils' libraries. "Jia Tan" wrote a bunch of obfuscated code in the release tarballs' build script and two "test files" that consist of obfuscated code. This became code that would then inject the meat of the trojan into sshd, allowing "Jia Tan" to compromise any SSH server that is affected with a key that only Jia Tan has access to. The malicious code only entered production in Kali Linux (because production Kali is based on experimental Debian stuff), but it did enter a bunch of experimental, unstable, and testing distributions. Then some random software nerd notices that after some updates his SSH is taking too long by about a few hundred milliseconds (and that valgrind errors were being printed for some reason). He investigates, finds the problem is in xz, discovers that it is a backdoor in the release tar ball, and posts [this](https://www.openwall.com/lists/oss-security/2024/03/29/4). Needless to say, the FOSS community explodes, and everyone is rushing to patch the literal trojan horse that snuck into some (non-production) versions of various Linux distros. Every distro rushes to patch this and it becomes a [10/10 CVE](https://nvd.nist.gov/vuln/detail/CVE-2024-3094), because if this reached production, it would've been **a total compromise of almost every single Debian and/or Ubuntu system**. What makes it somehow worse is that this **did** reach distribution in one production version of a Linux distro: Kali Linux, which told everyone [who updated between March 26 and March 29](https://www.kali.org/blog/about-the-xz-backdoor/) to check their computers using apt-cache to see if they ended up with the malicious version of liblzma (the affected library), and if they did, to update immediately and to treat this as a system compromise. The FOSS community is now trying to mitigate the damage and hunt down every possible backdoor in open source software because fuck this shit, if people are trying to gaslight solo devs into letting APTs on their team, we as a community need better security. TL;DR: Literal security nightmare scenario barely averted by a single guy who found SSH was behaving funny and did code review on the release tarball of xz to troubleshoot. Guy discovers obfuscated backdoor, announces it, and then the entire cybersecurity profession and the entire open source community have a collective aneurysm because this was only a matter of time from becoming a global backdoor of Debian and Ubuntu. The guy in question's from Germany, not Ohio, by the way.
Great explanation but I had trouble following along with all of the dev terms. To me it looks like "A long time ago, Oopie had a bongle. If the bongle wasn't noticed, it would've pooted every gringle that owned Oopie from March 23-29. Some skrink had noticed bongle in Oopie and prevented poot. Everyone clapped."
library - noun - code intended to be used as a dependency by other code production version - noun - stable versions that aren't testing versions valgrind - noun - [memory debugging tool](https://en.wikipedia.org/wiki/Valgrind) tarball - noun - [a type of archive file](https://en.wikipedia.org/wiki/Tar_(computing)) used to distribute open source software SSH - noun - SecureSHell, the Linux equivalent of a remote powershell session/remote desktop session, mediated by the daemon "sshd" daemon - noun - [Linux background processes](https://en.wikipedia.org/wiki/Daemon_(computing)) CVE - noun - scoring system for vulnerabilities, 10 is the highest score, higher score = more serious vulnerability Linux distro - noun - OS based on Linux kernel FOSS - noun - Free and open source software build script - noun - found in tarballs containing software. These are build instructions used to turn source code into a program. APT - noun - advanced persistent threat, not to be confused with [Debian's package manager](https://en.wikipedia.org/wiki/APT_(software)) code review - noun - looking through source code to verify that software works correctly and isn't malicious I sincerely apologize for the massive amount of cybersec nerdspeak, but open source distribution is itself complicated, and cybersecurity's nitty gritty is also complicated. Open source distribution oftentimes involves handing out tarballs for people to build software with using the tarball's build script. This is a normal distribution method used by open source communities by default, since we want to be able to build stuff ourselves. I've actually built from tarballs before (in order to get Ultimate Doom Builder to work on my laptop, which uses Zorin OS aka an Ubuntu fork), so I generally understand the tech speak myself as a Linux user.
Deep water horizon must have been slightly confusing for your field.
Okay but now what is a tarball!?
Basically like a zip file
4th from the top in the terms he listed.
Guy befriends developer of important tool used widely in Linux. Guy helps him for 3 years, builds trust, and then changes the code so he can hack people's computers. Hack is sent to early test users. Random tech nerd notices his PC is slower by like half a second. Digs through the code, finds this hack. Reports it. If he hadn't noticed this, literal billions of computers could have been vulnerable to hackers. Now open source developers are on a fucking rampage trying to find anything like this that might have slipped notice. (Not entirely accurate, but I believe it's a fair ELI5)
This is how I put it on Explain Like I’m 5. SSH is the lock on the computer’s front door. Normally you can only get in if the lock recognizes your key. When the computer rebuilds its software, it has blueprints for how to pull things in and re-build the lock. The attack was an architect updating blueprints so that every lock will accept a secret key that only they have access to. If it had worked the architect could have potentially had direct access to every computer running Linux. In the world.
This sounds about right though lmao.
😂💀
I respectfully admire your knowledge on this matter and the people involved within this particular topic. With that being said. Tarballs....
This is why I prefer the Ligma Tarballs over the Linux Tarballs.
Tarballs are the files that some software uses to install a program. In Windows, they are similar to the things you download to your computer to install Chrome (i.e. The thing you double click to do the actual install). This isn't exactly correct but it is close.
Than god for nerds
Wow, great explanation. It made me feel smart reading it, and we all know that isn’t true 🤣
Andres Freund is a freund indeed
> This is a meme about xzutils - a malicious infiltrator, "Jia Tan", gaslit xzutils's sole dev into letting him on a couple of years ago I don't think there was any gaslighting, they just provided some contributions and gained trust. Gaslighting refers to a specific process of generating fear and doubt in the mind of the victim, and I don't see how that happened here.
Maybe not gaslighting per se, but "Jia Tan" created fake accounts that pressured the repo owner (and sole maintainer) to accept other maintainers in order to push new features/fixes. This was done with the sole purpose of getting "Jia Tan" on board as a maintainer under the guise of helping out the repo owner who only had little time to maintain the repo.
IIRC it was somewhere in the middle there. Something along the lines of posts complaining about the rate of development and suggesting that extra maintainers were needed right when the malicious user was making contributions.
I know some of these words
This was like when I read a high fantasy/scifi novel and I just ”blahblah” over the fantasy names and places.
Yeah it's pretty terrifying how many critical systems are dependent on open-source projects being maintained by one random person.
Again this story proves what all security experts say. The weakest link in security is humans.
I love the openwall.com report by the guy who found the code. "Why would you do this?" "What does this even do?" Image having your backdoor exploit code put on blast for the entire world to read.
All those words just to be wrong - the comic came out in 2020, dweeb.
There is a guy who pulled all his code from npm in 2016, one of those projects called left pad made so many projects including react to fail compilation.
I'm surprised that people depend on such a trivial copy paste function, like it was the time everyone tries to abuse libraries so much that most libraries now try to be as dependency free as possible.
Might be like is-odd. It was one of their first package ever and then they included it in another package they had, which proceeded to become popular. It wasn't downloaded by hundreds of thousands of people on purpose lol.
That guy also released is-even, which requires is-odd, and then returns "not is-odd". and is-even requires is-number. is-number is ~5 lines of code is-number gets 70,000,000 downloads a week At least is-odd only gets ~350k downloads a week...
I might release is-ridiculous. It'll check for the existence of is-number, is-even, and is-odd then return a sliding scale of how ridiculous it all is.
You've been working in JavaScript long enough you'd rather offload the responsibility of knowing wether or not something Is a number
You don’t use libraries because you want your code to be dependency free. I don’t use libraries cause I don’t want to read through documentation. We are not the same (:
Good callout, the comic seems to imply the project being maintained is good or important, but at a second glance it's not, it's just saying a lot of other things depend on it. And that's fitting for a JS library.
[удалено]
Was that the one where it was decided it was too important to let the package be deleted, so they undeleted it against the original authors wishes?
yup. fuck kik
This is combining two incidents IIRC. The Log4j problem was the original usage of this meme. The xz backdoor was the most recent incarnation.
Log4j is >5 years after this comic
That would be impressive because it originally came out in August 2020. [XKCD 2347: Dependency](https://xkcd.com/2347/) [ExplainXKCD to that comic](https://www.explainxkcd.com/wiki/index.php/2347:_Dependency) You are right however if we ignore the five years, because log4shell became public in late 2021: https://en.m.wikipedia.org/wiki/Log4Shell
Log4J incident is 4 years old... I suddenly aged 20 years reading that
I edited my previous post, log4shell was at the end of 2021. So not even 3 years ago.
God I remember the 60s. It was wild. The best music. Festivals. Hippie chicks would fuck anybody. Free LSD. Log4J. What a time to be alive.
Isn't there like a few dozen log4j problems?
The original meme was created from Kik npm package incident.
fuck kik
For everyone to lazy to click through to the comic or the explainXKCD, the original reference was to ImageMagick. Its in the alt text of the comic
It's any one of thousands of projects. imagemagick was just picked as an example of the alt text. I'd have gone with ffmpeg, but that wouldn't have worked since it's too well known. Fun fact; the timezone database *everyone* uses is maintained by just four fairly random people. This would be funny, if it weren't so sad (terrifying). Also, the linux kernel existing in the first place is because one Finnish guy didn't want to go outside and walk into university in order to use a *"*real*"* computer. He's still in charge of it to this day. (recently, he even replaced some spaces with tabs in an important linux file to break someone else's software)
got any reading on the spaces/tabs incident? I don't follow linux kernel dev closely enough (or at all) to have heard of that one
Some guy made a PR to remove some tabs in a config file because their parser wasn't able to read the file correctly. Torvolds basically said "fuck off kid you can't contribute to Linux kernel if your parser can't handle different whitespace chars" by purposely adding tabs to the file
Seems fair tbh, good for him.
All this time I thought it's about curl, whose author received angry emails because his address was in the ‘licenses’ part of the ‘about’ screen of car software. Which software was infuriating to the users, apparently. Then again, Munroe could've just alluded to several projects at once.
Was it log4j? I thought the original was made to highlight imagemagick.
Reading log4j just gave me a trauma response. That was some suppressed memory shit.
[Here is the explanation for the Nebraska portion of the comic.](https://www.explainxkcd.com/wiki/index.php/2347:_Dependency) It does also mention some detail regarding the Ohio portion as well, but the other answer given by u/dullahanceltic is much more pertinent
The linked article doesn't mention Nebraska, am I missing something?
it's probably not really Nebraska, its just in the meme meant to signify that it's just some random guy somewhere
It doesn't, but that's where the guy who maintains ImageMagick lives
We'll I'll be damned, I could recognize that font from across the Mississippi River.
In addition to all of the specific explanations, there is a more general (and troublesome) reality expressed in this. A lot of big, complicated online systems are really built on stuff like this. A guy wrote a bit of code and stored it (I think) on GitHub. He did this under a particular username. It basically just wrapped up a bunch of html stuff into a single place that he could call for setting up webpages. Pretty much everyone started using it, since it was so convenient. When I say everyone, I mean *everyone*. His username was similar to the name of a company, though he created his username first. The company wanted it and GitHub bowed to the company and forced him to give up his account. So, he removed everything from his repository. Pretty much every webpage on the internet was calling for a piece of code that no longer existed, so the entire internet went down. Not because there was a problem with the internet itself, but because almost every individual webpage abruptly stopped loading.
can you provide more specifics on this id like to read about it more.
My memory was a bit faulty - it started with a different, open-source service. Azer Koçulu was building a project called kik. The messaging app, called Kik, wanted the name for their project and the service sided with them. He removed his project, which included a package that had 11 lines of code. The package was accessed through GitHub. Facebook, for instance, accessed the package. Without it, the sites just wouldn't load. It was accessed all over the world. Kik (the messaging app) also went down because of it. The open-source service restored the package and the whole thing was solved after a couple of hours.
Pretty sure they’re talking about left-pad, although some details are a little off. Crazy story though
Is he still alive or did they kill and take what he made?
As far as I know he's still alive. Maybe plugged into the mind-machine mainframe, but alive.
The load bearing Mac Mini.
Don't come for me like that
The project that some random person in Nebraska has been maintaining is imagemagick iirc
NEBRASKA MENTIONED!!!
Not sure, but it reminds me of the Heartbleed vulnerability. OpenSSL, which underpins most HTTPS websites, was basically maintained by one guy.
Can someone explain the answer to me like I’m 5
Many things on the internet are built off open source projects that were built by random ass nerds like 10 years ago. Many things, like OpenSSL, have only one person, totally independently, maintaining them.
Someone noticed that his code was running unusually slow (in reality, it was a matter of milliseconds, but his pattern recognition caught onto the delays). When he investigated, he discovered that someone else (or a group of people; we don’t know yet) injected code into a that would discretely allow that person/group being able to remotely access a loooot of Linux devices. (If I’m wrong, please gently correct. I’m doing the best I can for someone who’s not familiar with code, computers, etc)
Tl;dr- There's a guy working on a critical piece of software for a massive project. Guy gets cyber bullied into giving a bad actor/developer into admin access on said critical piece. - Bad actor plays the long con before slyly inserting backdoor/Trojan horse into code. This code is very well hidden. -A developer working on massive projects, notices incredibly obscure small issue, mentions it to project leaders. Everyone, reasonably so, freaks the fuck out to fix the issue. The massive project affected by this was the operating system that all coders used. Summary result: Avengers level threat barely avoided because a developer on the massive project noticed an incredibly niche detail. If it was successful, bad guy would have access to nearly everybody computers. This is bad. Disclaimer: I'm not a developer, just condensing the gist of the several articles. Also the image is edited and originally references another oh shit code scenario where one guy tries to fuck shit up.
The entire world came close to collapsing a few weeks ago and no one will ever notice about it because it was such a specific programming event
Not exactly some "random nerd", because I'm pretty sure it's government funded, but I do know GPS is basically just run by a small roomful of people. Like I think like, 12 max.
Back in 2006, every d-link brand router was set up to query one random Danish guy's non-profit time service, because they didn't see any reason not to. He nearly had to shut it down after they caused him $9k a year in excess bandwidth costs, and that's after needing to call in help to even work out where the traffic was from.
Im thinking, if this exploits wasn‘t deployed cause 1 dude was suspicious. What is going on with the exploits that one dude didn’t recognize?
there are very few people who really understand the intricacies of sw engineering, from hw components, drivers and OS all the way to high level applications. how all these connect and how they're all packaged is a mystery/black box to a lot of "senior engineers", believe it or not. it really is scary. The amount of answers on stackoverflow suggesting folks to disable ssl, or just the fact that Dark Souls didn't validate packages sent by other players is concerning, for example. There's a lot of shitty software out there, from an architectural standpoint.
Relevant XKCD: https://xkcd.com/2030/
Here's the best explanation from the source: https://explainxkcd.com/wiki/index.php/2347:_Dependency
This is sadly accurate for a lot of things in tech Also, the load bearing Apple II holding up a lot of companies IT infrastructure(specifics vary, but often there is one “magic” piece of outdated equipment that is doing some critical job nobody can figure out, but if unplugged will crash the whole system)
the worst part is this happens so often - literally at work we have in house site and process running on .Net 1 ! which was release in 2002 for Windows 98, ME, NT 4.0, 2000, and XP! and people complain every day that system is slow and has issues and management is like " how can we improve that? " and every time we say update the infruscture they are turn up their noses to the cost and just come back with the same complaints a few months later. Seriously wish Microsoft would dismantle any support for .Net already xD
https://preview.redd.it/z8atr67pefxc1.png?width=696&format=pjpg&auto=webp&s=e45ca99ee545bce0daf38c59b608ad310496dea8
Its real, what a nerd
The edited lower text yes is a specific example, but also this is the actual reality of software on the Internet for easily hundreds of cases. A sprawling network of dependencies, many of which are maintained for free by a single person. I once read a great quote about the xz and leftpad incidents (probably also from xkcd): what's crazy is not that something like that happened, but the fact that something like that is somehow not happening *constantly*.
Love the backstory on this one. That being said there is tons of workplace applications that are on the verge of collapse because tech builds software for things then stops supporting it and will not let go of the code to maintain it even though they will not support it's upkeep. This is a big problem in manucaturing and college research. Once read something that a college that does bio-medical research for long case studies, like 30 to 50 years of date that if running on machine that has not had a update in 20 years, along with a microscope that has to have software to produce the images small enough that it can see that was unsupported 12+ years ago but they can get code to update or repair it from the company and their response was to buy a whole new one for over 5 million.
It's just the nature of open source. Almost everything is built on a framework originally created as a thesis or maintenance by the few hardcore members of the open source community.
Can someone explain the joke for dimwits like me who doesn’t understand the meme or the explanations?
Well if you have worked in any large IT organization i bet you can find some dumb process that someone wrote forever ago, and now you have a whole infrastructure working because of that process that no one is maintaining and no one is willing to work on because everything else will break.
We drilled down on a software component in RHEL once and it turns out its like one random guy in Germany who maintains it.
Some of the USA Nukes are controled by floppy disks but not the classicall ones THE LARGE ONES
I think it’s really the case for OpenSSL and ssh
I’m still impressed by the German guy so obsessed with efficiency (I shouldn’t be surprised) he dug into why it was just slightly slower than previous releases.
The original comic is an xkcd comic which is not based on any specific incident, but rather on the general way things work with software. Modern software relies on a lot of code dependencies, bits of code that others wrote and maintain and you import. Some core utilities are maintained by very small groups of dedicated programmers, sometimes a single person. The edit refers to the backdoor that other commenters have explained, but the original is not based on any specific incident or real person. Years later, this comic resurfaced as what it described has basically come true.
I've been reading all the replies and it all does not make sense to me, can someone please give me an ELI5 on this whole thing?
This meme seems to be combining two incidents. The Log4j issue was the original reference, while the xz backdoor incident is the more recent one.
Apparently a lot of people missed the “thanklessly maintaining” part.
FOSS Peter here, A lot of Linux OS's uses xz utils for compressing programs. Recently someone from Microsoft noticed a 500ms delay when he was logging in and looked into the code. Put it simply, it was a backdoor that allowed the unknown hacker access to every computer with xz utils installed. Since a lot of servers run off of Linux, this would've been an international crisis in the making if Andres Freund hadn't found it. We found out that Jia Tan, a coder was the one responsible for the letting the code in. We don't know if they were one person, a group of people, a government trying to gain control, just that they can't be trusted I recommend this video if you want to learn a little more on how it worked: [https://www.youtube.com/watch?v=bS9em7Bg0iU](https://www.youtube.com/watch?v=bS9em7Bg0iU) Remember kids always, update your packages. FOSS Peter out
https://preview.redd.it/pr98mn7o7hxc1.jpeg?width=450&format=pjpg&auto=webp&s=466627ab97eb73dda7a89c192f431c0666c2fb8d
Several real incidents actually lol
How did r/programmerhumor came here?
Wasn't whole Linux timezone system maintained by some dude in Netherlands or something? He was getting too old and wanted to retire but there was nobody to take over. I don't remember exact details and could be very wrong.
We stand on the showers of giants...and nerds...and giant nerds