Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
An attack that hides malicious code in something you install or build with, such as a package, container image or build tool, so it runs with your permissions even though your own code is fine.
In short
A software supply chain attack is when an attacker gets malicious code into something you install or build with, such as a package, a container image or a build tool, instead of attacking your own code. You install it the normal way, and it runs with your permissions. On 21 October 2026, a poisoned release of the popular Python package LiteLLM (version 1.107.0) stole cloud keys, SSH keys and AI provider keys from machines that installed it.
In short
A software supply chain attack is when an attacker gets malicious code into something you install or build with, such as a package, a container image or a build tool, instead of attacking your own code. You install it the normal way, and it runs with your permissions. On 21 October 2026, a poisoned release of the popular Python package LiteLLM (version 1.107.0) stole cloud keys, SSH keys and AI provider keys from machines that installed it.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Software Supply Chain Attack across a run of lessons, not one page. These 4 alone are about 292 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Your LLM app loads weights, datasets, packages, and MCP servers you never wrote. Pin, hash, scan, sign, and verify provenance so an import cannot own you.
What does an unpinned requirements file install six months later? I asked uv what my five-line file would have installed at nine dates over two years. Every three months, 4 to 9 packages moved; six months apart, 6 to 9. At lesson 3's own dates the file gave lesson 3's exact venvs, and all 6 models saved after one install failed after a later one. A hash-locked file installed the same 9 packages twice and trained the same model to the last bit. A changed wheel under the lock stopped both uv and pip, but an edited hash in the lock only made them build scikit-learn from source, with build tools from outside the lock. With --no-build, that edit stopped the install too.
How do you know the model file you load is the file you trained? I recorded the features chapter's model's sha256 and 100 golden rows at training time, then changed copies of the file five ways. A sha256 check caught all of them, and also fired on a harmless 4-thread rebuild that changed one byte and no prediction. One flipped byte in twenty crashed the process; another sent a branch outside its tree and gave garbage scores in the lab, then looped or crashed in later runs. Changing one rare leaf moved 40 of the top fifth of customers, and 11 of 20 sets of 100 golden rows missed it; by the exact odds, 100 random rows catch it 64% of the time and 300 rows 95%. The scanners saw no difference at all. A signature, made with model-signing and a throwaway key, refused every copy and told apart who had signed.
What makes a model's container image big, and what does slimming cost? I built real Docker images of the features chapter's model: the full python:3.13 base, slim, slim with a hash lock and no pip cache, a multi-stage build, and alpine. Unpacked, they measured 1,569.1, 521.9, 460.8 and 464.2 MB; alpine did not build, because scikit-learn has no musl wheel. 77% of the full image was its base. All four images predicted the same numbers as each other, and 33 of 26,851 differed from the Mac in the last binary digit, with test AP equal to the bit.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with AI Supply Chain Security: What You Import Runs as Youpay once,
yours for life
Think of a restaurant. The chef is careful, the kitchen is clean, but one supplier's flour is contaminated. Every dish made with it is bad, and the chef did nothing wrong. Software works the same way.
Your app is built from your own code plus hundreds of pieces you did not write: open-source packages, a base container image, build tools, CI actions. CI (continuous integration) is the automated system that builds and tests your code. Each of these comes to you through a chain of people and machines: the maintainers who write it, the source repository, the release build, and the registry you download from, such as PyPI for Python, npm for JavaScript or Docker Hub for images.
A supply chain attack breaks into one link of that chain. Everyone downstream then installs the bad version in good faith.
LiteLLM is a widely used Python library and LLM gateway that lets apps call many AI model providers through one interface. Here is what happened, from the maintainers' incident post, their GitHub updates, and SafeDep's analysis.
Notice what did not help. The code review on GitHub was fine, because the bad code was never in the reviewed source. A version range like litellm>=1.100 helped the attacker, because it pulled the newest release automatically.
No single control stops every case. Each one below shrinks the window or the blast radius, meaning how much one bad package can reach.
Interviewers ask about this in security, platform and AI engineering rounds: "a package we depend on was compromised this morning; walk me through the next hour". A strong answer separates the steps: find every place it was installed (the inventory), stop the bleeding (pin a clean version, block the bad domain), assume the secrets are gone and rotate them, then look for the backdoor process and anything it downloaded.
AI apps add more to the chain: model weights, datasets, tokenizer files and tool servers, all loaded by code you did not write. The AI supply chain security lesson in the AI Engineering course covers pinning, hashing, scanning and signing for all of them, and the library version skew lesson shows how an unpinned upgrade breaks a model quietly even when nothing is malicious.
PyPI
The Python package index. Removed litellm 1.107.0 the same day, and supports Trusted Publishing and signed attestations for releases.
GitHub
Where the LiteLLM source lived. The attacker used stolen GitHub login tokens to run the release workflow against a lookalike repository.
Docker Hub
The latest and main-stable image tags pointed at the poisoned build for a few hours, which is why pulling by digest matters.
uv
A Python package manager with lockfiles that record hashes, and an exclude-newer setting that ignores releases newer than a chosen date.
It is an attack on something you install rather than on your own code. The attacker gets bad code into a package, image or tool, and you install it the normal way, so it runs with your permissions and can steal your keys.
On 21 October 2026, LiteLLM version 1.107.0 on PyPI contained a credential stealer. It ran every time Python started and collected cloud keys, SSH keys and AI provider keys. Older examples are the xz Utils backdoor found in 2024 and the SolarWinds Orion compromise found in 2020.
Pin exact versions with hashes, wait a few days before taking new releases, pull images by digest, check provenance where the registry offers it, keep long-lived secrets off build machines, limit outbound network traffic, and keep an inventory of what you run so you can respond fast.
It protects you from new bad releases, as long as you pin before the bad one appears and check the file hash. It does not help if you pinned the bad version itself, so pair pinning with a short cooldown before upgrading.
Secret Management
Securely storing and distributing credentials, API keys, and certificates. Tools like Vault, AWS Secrets Manager, and SOPS prevent secrets from leaking into code or logs.
CI/CD
Continuous Integration and Continuous Deployment: automating the process of testing and deploying code. Push code, tests run, and it ships to production automatically.
LLM Gateway
One server between your apps and every AI model provider. Apps call it in one format; it checks keys and budgets, picks a model, falls back on failure, records cost, and holds the real provider API keys.
Zero Trust
A security model that never trusts any request by default, even from inside the network. Every request must be authenticated, authorized, and encrypted regardless of origin.
API Key
A simple token passed with API requests to identify the calling project or application. Not a substitute for user authentication but useful for rate limiting and usage tracking.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
This page explains the idea. The "AI Supply Chain Security: What You Import Runs as You" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the AI Supply Chain Security: What You Import Runs as You lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about Software Supply Chain Attack in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
