[Paper Review] An LLM-based Quantitative Framework for Evaluating High-Stealthy Backdoor Risks in OSS Supply Chains

The xz-utils compromise is the main context for this paper, so it is worth laying out clearly.

xz-utils is a widely used compression package on Linux, and its core library is called liblzma. On March 29, 2024, PostgreSQL developer Andres Freund was investigating unusually high CPU usage during SSH logins and Valgrind errors on Debian sid. He eventually traced the problem not to Debian, but to the 5.6.0 and 5.6.1 release tarballs published by the xz upstream project. The incident was later assigned CVE-2024-3094, and Freund published the initial technical details on the oss-security mailing list.

This was not a conspicuous block of malicious source code committed to Git. The M4 macro that triggered the build existed only in the release tarballs, while the actual object file was disguised as test data in the repository. Under specific build conditions, a script unpacked that object and linked it into liblzma. OpenSSH does not link directly against liblzma, but in affected Linux configurations, systemd brought the library into the sshd process. The malicious code could then interfere with the pre-authentication RSA verification path, creating a risk of unauthorized access or remote code execution.

Fortunately, when the backdoor was discovered, versions 5.6.0 and 5.6.1 were still mostly confined to development branches such as Debian sid and Fedora Rawhide, rather than widely deployed stable releases. What made the attack frightening was not just the backdoor itself. It was the way the attacker chained together a malicious object, test files, release tarballs, and downstream build conditions. If you only reviewed the Git diff, you could easily miss half of the mechanism.

After the incident came to light, a common reaction in the security community was that a long-running infiltration like this was too rare to model.

I found that argument reasonable at first. But if a red team accepts it, target selection becomes little more than luck.

This AAAI 2026 paper asks a different question: instead of guessing which library will become the next xz victim, stand on the attacker’s side and ask which upstream projects are worth a long-term investment. The scope is deliberately narrow. It does not scan npm or PyPI; it looks only at APT upstream projects that Linux distributions may actually ship.

The framework then asks four things: how far the impact can spread, how easily a payload can be hidden, how hard it is to gain trust in the community, and whether the build pipeline offers a way to trigger the payload.

The open-source implementation is called HSBRiskEvaluator. I read the paper alongside the repository. The rest of this article follows the attack process.

Target upstream projects that distributions actually ship

The first filter is not GitHub stars, and it is not the trending page.

Debian has 319 packages marked required, important, or standard; 66 of them have public GitHub repositories. The evaluation covers those 66. The paper is explicit about the reason: a package without a public collaboration surface cannot be scored on the community dimension.

The batch-processing scripts in the repository follow the same path:

1. get_priority_packages.py extracts high-priority packages from APT
2. The pipeline expands their dependencies and writes them to YAML
3. An LLM maps downstream mirrors on salsa.debian.org back to their GitHub or GitLab upstreams
4. It collects pull requests, commits, workflows, and binary-file paths
5. It scores four dimensions and combines them into HSBR

This also explains why the original installation supports Debian-based systems only. Dependency Impact queries the local APT database directly: is the package essential, and how many high-priority packages depend on it? Moving the framework to Fedora or npm would require more than renaming a package. The measuring stick for this entire dimension would need to be rebuilt.

xz did not become dangerous because it had enough stars. liblzma was shipped by distributions and later intersected with OpenSSH. That is what made the incident serious. APT’s priority and essential labels are a way to measure that blast radius.

Tools such as OpenSSF Scorecard are closer to repository hygiene checks: are dependencies current, is CI configured safely, and are artifacts clean? This paper adds two questions that sound much more like an attacker’s concerns: how many pull requests does it take to become a maintainer, and does the repository already contain binary files under its test directories?

Where the four scores come from

The paper breaks a poisoning campaign into four stages, with one score for each stage:

The paper combines the dimensions with this formula:

R_total = 0.3·DI + 0.2·PC + 0.3·CQ + 0.2·CI

DI and CQ receive the highest weights. In the xz case, the logic is straightforward: the package needs to travel far, and the attacker needs a way into the project.

The code and paper do not line up perfectly here. The repository’s calculator.py swaps the 0.3 and 0.2 weights assigned to CI and CQ. I use the weights stated in the paper below.

The phrase LLM-based in the title can easily suggest that the model reads an entire repository and hunts for backdoors. It does nothing that dramatic. Most of the metrics are static counts: APT labels, the proportion of direct pushes, the number of PRs submitted before someone became a maintainer, and whether Actions in workflow files are pinned to 40-character SHAs.

The LLM fills four gaps that are awkward for static analysis:

• Map packages to upstreams: match Debian package names to GitHub or GitLab; the model is gpt-4.1:online
• Classify binary paths: inspect path strings, not file contents, to decide whether a file belongs to tests/, docs/, or src/
• Check whether a PR matches its stated purpose: compare the title and body with changed_files, up to 300 files in batches of 30, without reading the diff
• Identify injectable CI YAML: look for patterns such as pull_request_target or interpolation of a PR body into run:

There is another implementation difference: the paper reports experiments with GPT-4o, while the repository defaults to openai/gpt-4.1-mini through OpenRouter. Anyone reproducing the experiment should not collapse those into a generic statement that it “used GPT.”

Score 66 packages, then check the answer against xz

Across the 66 high-priority GitHub repositories, the four dimensions do not all turn red at once. Each dimension has a different shape:

• CQ is the broadest problem: only 1.5% of repositories are high risk, but more than 80% fall into medium risk. The door is not wide open, yet entering most projects is not prohibitively difficult either.
• CI has the most concentrated high risk: 39.4% are rated high risk. Disabled Dependabot appears in 62.12%, untrusted Actions in 39.39%, and Actions not pinned to commits in 31.82%.
• PC is not a fringe condition: 13.6% are high risk, and 34.85% of projects contain binaries in test directories.
• DI looks comparatively mild: more than 78% are rated low risk. That does not mean their dependencies are unimportant. The sample already consists of high-priority packages, so differences in downstream impact are compressed within the group.

The five highest single-metric high-risk rates are:

I did not fully understand the Community Review result. Its high-risk rate is 0.00%, yet the paper discusses it in the same section that calls CQ the broadest attack surface. The grading thresholds may explain the difference, but the paper does not spell that out. I would rather leave the question open than answer it for the authors.

The paper then replays xz. It does not retell the incident; instead, it feeds the events that already happened back into the four dimensions to see whether the indicators line up:

I also checked the raw tukaani-project/xz output in the README: binaries in the test directory are marked true, the repository contains 95 binaries, Dependabot is disabled, and the untrusted Action ratio is 0.78. The maintainer distribution even includes contributors who gained access after submitting zero PRs. These signals match the values reported in the paper.