It was luck. Pure luck.
As the world went about its normal business last week, none of us had any idea we were staring down the barrel of a doomsday scenario.
Just last week, a software engineer discovered something amiss in a software tool used in the Linux open source software operating system.
As he was testing the software tool, he noticed a delay of 500 milliseconds, which prompted him to investigate…
To normal users like us, we wouldn’t even notice 500 milliseconds of delay. It’s only half a second.
But the engineer, named Andres Freund, was benchmarking the software performance…
And something wasn’t right.
Last Friday, Freund posted the following:
“I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU.” — Andres Freund
The comment might not sound like much, and it might not make much sense.
But it turned out to be a massive deal.
Something was consuming an unusual amount of computer processing power, which was causing the software to slow down by a half a second.
It got worse.
Hours later, he posted the following, having discovered what had been done:
“The upstream xz repository and the xz tarballs have been backdoored.”
Gibberish to most of us. But it meant that the open-source software had been compromised.
Again, that might not sound like a disaster. After all, software is compromised every day around the world in thousands of instances of cyberattacks. Surely, this wasn’t a big deal…
But it was.
That’s because the software was about to be deployed on every internet server running Fedora, Ubuntu, or Debian — all versions of Linux.
And here’s the kicker…
96.3% of the top million web servers run Linux.
Just about every software service that we use daily is running on a Linux-based server. Google, Facebook, Instagram — you name it.
And all the major cloud service providers that run the world’s cloud-based software programs all use Linux, as well — like Google Cloud, Amazon Web Services, and even Microsoft Azure.
What this means is that, with a “backdoor” installed into 96% of the top 1 million web servers, the “actor” behind the backdoor could monitor global web communications…
Not only that, it could have taken the entire internet down.
Lights out. A global internet “crash.”
Fortunately, Freund found the backdoor and averted the global disaster...
But I still can’t get over the scale of the damage that could have been done, had the software been released to the world’s Linux systems.
And the story gets even more interesting…
The software backdoor has been connected to one Jia Tan, a pseudonymous (not a real name) software developer that had been an active contributor to the Linux open source community since 2021.
In the tech community, it’s not unusual to use pseudonyms, especially in software. So that wasn’t a red flag…
Jia Tan, whoever the person is, worked for more than two years to build trust within the software community… to the point where he was allowed to make changes to the software code. [Note: Open source projects tend to be tightly controlled so that only “trusted” contributors can actually update the software code.]
It was premeditated and planned out over years.
The prevailing belief is that the backdoor efforts were backed by a state actor. It was sophisticated and took a lot of time and money.
There was clearly an agenda at play. Was it China, North Korea, Russia, Iran… or someone else?
These kinds of backdoors to access internet communications are not new. One of the most widely known backdoors is tied to China-based telecommunications giant Huawei.
More than 20 years ago, Huawei stole Cisco’s IOS software, which is the software that runs Cisco’s internet routers, which largely power the world’s communications infrastructure.
Cisco had spent billions in research and development. Huawei took it, reverse engineered Cisco’s hardware, and vaulted onto the world’s stage with dramatically cheaper routers that worked just as good as Cisco’s.
As Huawei grew exponentially, it was able to use the cash flow to pay for changing the code and developing its own. But it had a head start with Cisco’s code, an advantage that no other company in the industry had.
It also installed backdoors into its telecommunications equipment, sold around the world, which allowed Huawei to eavesdrop on highly sensitive internet and mobile communications.
Countries around the world aggressively purchased Huawei, and also ZTE’s (also China-based) telecommunications equipment, because it performed well and was far cheaper than Cisco.
But over the last few years, the extent of the software compromises became very well-known, resulting in several major bans:
The list goes on.
Clearly the security problem was so severe, and created such an intelligence risk, that the equipment had to be banned outright.
And yet, the scale of the backdoor that was just discovered is so much greater than that of Huawei and ZTE…
Gaining control of the million most-important web servers on the planet is a far greater prize than the control China got through Huawei and ZTE, with their telecommunications equipment backdoors.
The reality is that World War III has already started. Sadly, it’s all around us. It’s not what the world is used to.
It’s largely driven by highly sophisticated, multi-year cyberattacks that usually require forms of social engineering — effectively software engineers who are spies with a very clear mission to hack into critical software code.
They’re also equally sophisticated in psychological operations (psyops) — designed to foment hatred and division amongst populations, as well as weaken their ability to think for themselves.
It is highly effective, and sadly, we can already see the effects spreading and working in the greater population.
But all of this raises a critical question…
If the entity behind Jia Tan got this close to gaining control of the world’s internet servers, how many other pieces have been compromised that we don’t know about yet?
We got lucky this time, thanks to one software engineer that just had a hunch.
But logic would dictate that the worse it yet to come.
Malicious code has already been deployed in the world’s software-controlled systems, we just haven’t felt the repercussions yet.
Questions, comments, or feedback? We always welcome them. We read every email and address the most common threads in the Friday AMA. Please write to us here.
