Explainedllama.cppggmllocal LLMLLM inferenceKV cachequantizationHugging Face

llama.cpp 1.0 explained: what the first stable version changes, the new FP8 KV cache, and how to upgrade

llama.cpp, the open-source engine that runs large language models on laptops, phones and servers, reached version 1.0.0 on 13 November 2026. The big change is a promise: from now on a breaking change to its C API means a new major version. The release also adds an FP8 KV cache, which halved the memory of our test cache, and tightens the security of llama-server.

This explainer covers one piece of news. The AI Engineering course teaches how systems like this work, step by step.

This page's course

The AI Engineering course

₹1,333 / $35 once, with lifetime access.

Start the first free lesson See the course

Best value

Both courses together

₹1,999 / $49 once, for System Design and AI Engineering, with lifetime access to both.

Get both courses

More ways to learn

All 9 interview packs: ₹3,000 / $49, every question with a full answer and a diagram, one payment.
System Design Interview Bootcamp: ₹15,000 / $300, self-paced, every class recorded.
AI Engineering Live: ₹15,000 / $300, live Saturday classes, and every class recorded.

Free PDF: the AI Engineering cheat sheet, 12 pages Get it free

Roni Das

8 min read

Published14 November 2026, 04:15 UTCUpdated14 November 2026, 04:15 UTC

Two courses by the author of this page:

Generative AI and Agentic AI

219 lessons · 10 free to read

₹1,333 in India · $35 elsewhere, once

Get Generative AI and Agentic AI

You own this course

System Design

770 lessons · 18 free to read

₹1,111 in India · $25 elsewhere, once

Get System Design

You own this course

Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor

Comparison of three KV cache types in llama.cpp 1.0, measured with Qwen3 0.6B and a 32,768-token context. f16: 3,584 MiB, 2 bytes per stored number, not new, started on the CPU and on the Apple GPU. FP8: 1,792 MiB, 1 byte per number, new in version 1.0.0, started on the CPU but not on the Apple GPU. q8_0: 1,904 MiB, about 1.06 bytes per number, not new, started on the CPU and on the Apple GPU. — The KV cache in llama.cpp 1.0: f16, the new FP8 type and q8_0, as we measured them

In short

llama.cpp 1.0.0 and ggml 1.0.0 were released on 13 November 2026 by the ggml-org team at Hugging Face, about three and a half years after the project began.
From now on a breaking change to the public C API means a new major version, a new feature or model a minor version, and a bug fix a patch version.
Version 1.0.0 removes 35 deprecated functions and the --defrag-thold flag, so old code and scripts can break when you upgrade.
It adds an FP8 KV cache (-ctk f8_e4m3 -ctv f8_e4m3). In our run it halved the cache, 3,584 MiB to 1,792 MiB, but would not start on the Apple GPU.
llama-server gets safer defaults: tools and private media URLs now need explicit opt-in flags, and CORS, DNS rebinding and SSRF problems were fixed.

Released: 13 November 2026
Version: llama.cpp 1.0.0, ggml 1.0.0
Previous: 0.9.0
Model architectures: more than 150
License: open source, MIT

This page is the free part.

The course goes deeper on running and sizing LLM inference

₹1,333 in India$35 everywhere elseonce, for the whole course

The Generative AI and Agentic AI course covers running and sizing LLM inference across a run of lessons, not one page. These 4 alone are about 201 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.

Quantization: The Same Model in Fewer Bits
The same Qwen2.5 3B model stored three ways and measured on a laptop: 6.18 GB at 16 bits per weight, 3.29 GB at 8.5 and 1.93 GB at about 5. The 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts and got the same 33 of 40 sums right; the 5-bit file kept 19 of 24, all 5 changes on near ties. Writing speed went from 11.1 to 18.3 to 28.4 tokens a second, while reading barely changed.
46 min · diagrams · code you run · quiz
The Context Window: What Happens When a Prompt Does Not Fit
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
46 min · diagrams · code you run · quiz
Attention: How Each Word Looks Back at the Words Before It
Attention lets every word in a prompt gather meaning from the words before it. Worked by hand with tiny numbers, then read from a real model, Qwen2.5-0.5B: a median of half of each word's attention went to the very first word, a word never looked ahead, and on 10 sentences where one word changes what 'it' means, the model was right 6 times and gave the same answer for both versions in 4 of 5 pairs.
49 min · diagrams · code you run · quiz
LLM Inference Optimization: Serving More Tokens Per GPU
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
60 min · diagrams · code you run · quiz

all part of

Generative AI and Agentic AI Engineering

219 lessons · about 230 hours · 10 free to read

₹1,333in India, by UPI

$35everywhere else, by PayPal

Get the Generative AI and Agentic AI course Read 10 lessons free first

You own this course

Continue with Quantization: The Same Model in Fewer Bits

pay once,
yours for life

What is llama.cpp 1.0?

“Moving forward, both llama.cpp and ggml will reflect API and ABI incompatibilities with major-version bumps.”

ggml-org, llama.cpp v1.0.0 on the Hugging Face blog, 13 November 2026

llama.cpp is a program, written in C and C++, that runs large language models on ordinary hardware: a laptop, a phone, a desktop GPU or a big server. It is what many local AI apps use underneath. Its maths engine is a separate library called ggml, which has a backend for almost every kind of chip, from Apple's GPUs to NVIDIA, AMD and Intel cards and plain CPUs.

Version 1.0.0 is less about one new feature and more about a promise. Many apps now build on llama.cpp's C library, and every time its interface changed, those apps could break. The team says it will now mark every such break with a new major version number, so a developer can tell from the number alone whether an upgrade is safe.

The Hugging Face post also points to Llama, a small desktop app for Mac and Windows from the same team. It runs open models on your computer, offers a chat window, and serves an OpenAI-compatible API on localhost:9931, so tools that expect OpenAI can talk to a local model instead.

What do the version numbers mean now?

llama.cpp used to ship numbered builds many times a day, and anyone building on it had to read the changes to know what had moved. From 1.0.0 the ggml-org projects follow semantic versioning, with these rules from the release post.

Plans and limits

Major, for example 1.x to 2.0: a breaking change to the public C API in ./include
Minor, for example 1.0 to 1.1: a new feature, backend update, model or API addition that does not break anything
Patch, for example 1.0.0 to 1.0.1: a bug fix with no API change

The release also lists the ways you can now install it: pre-built files for Linux, macOS, Windows, Android and iOS, plus Docker, Debian, Homebrew, conda-forge, MacPorts, Winget and an XCFramework for Apple apps. Nightly builds keep their own numbers; the 1.0.0 release matches nightly build b12359.

What else is new in llama.cpp 1.0.0?

The release notes are long. These are the changes most people will notice.

FP8 KV cache. A new cache type, f8_e4m3, stores the attention keys and values in one byte each instead of two. Turn it on with -ctk f8_e4m3 -ctv f8_e4m3.
MoE expert cache. For mixture-of-experts models, a cache in host memory for experts that do not fit on the GPU, with a setting for how much of the missed work the CPU does.
Safer llama-server. Tools without authentication and fetching media from private addresses now need the --allow-unauthenticated-tools and --media-allow-private-urls flags. CORS origins, DNS rebinding and a server-side request forgery hole in the CORS proxy were fixed, and the HF_TOKEN is only sent to huggingface.co.
New models. Kimi-K3.1-Pro, Ripple-65B-Preview, xing4_0 and the drama-3 text-to-speech model, plus MageVL-Cowork and Kimi-K3.1 vision.
Speech endpoint. llama-server adds an OpenAI-compatible /v1/audio/speech endpoint for text-to-speech models.
New web interface. The built-in chat page was redesigned, with chat groups, a markdown editor and better layouts on phones.

everything we teach

Two courses, interview packs and live classes, all by the author of this page.

self-paced course

AI Engineering course

Learn to build RAG apps, AI agents and evals step by step, with no heavy maths.

219 lessons, 10 free to read. Lifetime access.

₹1,333 once in India · $35 once elsewhere

You own this course

See the course

live classes

AI Engineering Live

Build one real AI support copilot from an empty folder, live with Roni Das, over 14 Saturdays.

Weekly on Saturdays, 8:30 PM IST. Recordings of past classes included.

₹15,000 full fee in India · $300 full fee elsewhere

See the live classes

self-paced course

System Design course

Learn how real systems are built and scaled, from your first database to a staff-level design interview.

770 lessons, 18 free to read. Lifetime access.

₹1,111 once in India · $25 once elsewhere

You own this course

See the course

interview prep

Interview prep packs

Real interview questions with a full answer and a diagram for each one, as PDFs you keep.

9 packs on sale. The one closest to this page is the Inference Engineering pack.

₹999 a pack, or all nine for ₹3,000 in India · $20 a pack, or all nine for $49 elsewhere

See the packs

Both courses together: ₹1,999 in India / $49 elsewhere, lifetime access.

System Design and AI Engineering in one payment. Bought one by one: ₹1,111 + ₹1,333 = ₹2,444, so you save ₹445. Bought one by one: $25 + $35 = $60, so you save $11.

Get both courses

Also open for enrolment: System Design Interview Bootcamp and Neo4j, GDS and GraphRAG. See all programmes

Hands-on: the FP8 KV cache, the new server checks and a removed flag

We upgraded llama.cpp from 0.9.0 to 1.0.0 with Homebrew on an Apple M4 laptop and ran one short script. It starts llama-server with a small model, Qwen3 0.6B, and a 32,768-token context, reads the KV cache size the server prints for three cache types, sends one chat request, checks the CORS reply, and then tries the same caches with the model on the Mac's GPU.

The KV cache is the model's memory of the conversation so far: for every token, every layer keeps a key and a value so it does not have to recompute them. Its size is easy to work out. This model has 28 layers and 8 key-value heads of 128 numbers each, so one token needs 28 times 1,024 numbers for keys and the same for values. For 32,768 tokens at 2 bytes each that is 3,584 MiB, exactly what the server printed.

Three things to take from this. FP8 halves the cache because each number takes one byte instead of two; q8_0 is close behind at about 1.06 bytes, because it stores a small scale with every block of 32 numbers. Second, a new cache type is only useful where your backend supports it: on our Mac, Metal could not write to an FP8 cache, so check your own hardware before you rely on it. Third, the CORS check works the way it should. For a request from another site, the server still names only the allowed page in its reply, so a browser will refuse it.

We also saw a new warning when the server starts with no API key and no CORS setting: "no API key is set and CORS allows all origins". If you run llama-server on a shared machine or network, set --api-key and --cors-origins.

What we measured: With llama.cpp 1.0.0 the FP8 KV cache took 1,792 MiB against 3,584 MiB for the default f16 cache, exactly half, and answered a chat request normally. With the model on the Apple GPU, the FP8 cache stopped the server at start-up, while q8_0 started and used 1,904 MiB.

$ MODEL=Qwen3-0.6B-Q8_0.gguf bash kv_cache_check.sh
version: 1.0.0 (build 12359, commit b1b770d90)

1. KV cache size by type (model on the CPU, -ngl 0)
f16      3584.00 MiB
f8_e4m3  1792.00 MiB
q8_0     1904.00 MiB

3. CORS preflight with --cors-origins http://localhost:3000
from http://localhost:3000 -> Access-Control-Allow-Origin: http://localhost:3000
from https://evil.example -> Access-Control-Allow-Origin: http://localhost:3000

4. The same caches with the model on the Apple GPU (Metal, the default on a Mac)
f16      started, 3584.00 MiB
f8_e4m3  server exited: pre-allocated tensor (cache_k_l0 (view)) in a buffer (MTL0) that cannot run the operation (SET_ROWS)
q8_0     started, 1904.00 MiB

5. A flag that 1.0.0 removed
llama-server --defrag-thold 0.1 -> error: invalid argument: --defrag-thold

Run on 14 November 2026 with the Homebrew build of llama.cpp 1.0.0. Shortened: section 2, the chat reply, is left out, and the q8_0 size is moved up beside the other two. The full output is saved next to the script. Download it from systemdesign.academy/labs/llama-cpp-v1/kv_cache_check.sh.

if this is what you came for

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

start here

Generative AI and Agentic AI Engineering

How models generate text: tokens, attention, the context window, the KV cache, quantization and serving, each explained from first principles with diagrams and a quiz.

219 lessons in 17 chapters: RAG, AI agents, evals, fine-tuning and serving
No heavy maths, everything taught step by step from the basics
127 lessons built on a lab that really ran
An AI tutor beside every lesson that answers and draws
Lifetime access, every new lesson included

₹1,333 once, by UPI

$35 once, by PayPal

$35 outside India

See the Generative AI and Agentic AI course Buy Generative AI and Agentic AI

You own this course Open it

Not sure yet? 10 lessons are free to read, no card needed.

How to upgrade to llama.cpp 1.0 without breaking anything

For most people who use the command line tools or an app built on llama.cpp, the upgrade is quiet. If you call the C library yourself or pass old flags in scripts, check these first.

Upgrade
On a Mac or Linux with Homebrew, run brew upgrade llama.cpp. Check the result with llama-server --version, which should print version 1.0.0.
Search your scripts for removed flags
--defrag-thold is gone and now stops llama-server with an invalid argument error. Remove it from start-up scripts and service files.
Rebuild against the new headers
35 deprecated functions and llama_set_warmup were removed from llama.h, and llama_set_embd_state_src is now llama_set_ctx_state_src. If your code still compiles, it does not use them.
Multimodal code
clip.h was removed and its work moved into the mtmd library. Code that included clip.h directly must move to mtmd.
Test a smaller KV cache
Try -ctk f8_e4m3 -ctv f8_e4m3 or q8_0 on your own hardware and model, compare the answers on your own prompts, and keep whichever works and stays accurate enough.
Lock down the server
If anything other than your own machine can reach llama-server, set --api-key and --cors-origins, and only add the new opt-in flags if you need them.

What does llama.cpp 1.0 mean for engineers?

A 1.0 version is a statement that other people's software now depends on this, and llama.cpp has earned it: it supports more than 150 model architectures, every mainstream method of speculative decoding, and almost every kind of hardware. For a team shipping a product on top of it, stable versions mean you can pin a major version and take minor and patch updates with confidence.

The FP8 cache is a reminder of where memory goes in local inference. On a laptop the model weights are fixed, but the KV cache grows with every token of context and every extra user. Halving it can mean twice the context, or twice the users, on the same machine, as long as quality holds and your backend supports it.

if this explainer was useful

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

start here

Generative AI and Agentic AI Engineering

How models generate text: tokens, attention, the context window, the KV cache, quantization and serving, each explained from first principles with diagrams and a quiz.

219 lessons in 17 chapters: RAG, AI agents, evals, fine-tuning and serving
No heavy maths, everything taught step by step from the basics
127 lessons built on a lab that really ran
An AI tutor beside every lesson that answers and draws
Lifetime access, every new lesson included

₹1,333 once, by UPI

$35 once, by PayPal

$35 outside India

See the Generative AI and Agentic AI course Buy Generative AI and Agentic AI

You own this course Open it

Not sure yet? 10 lessons are free to read, no card needed.

Free PDF · 12 pages

Get the free AI Engineering Cheat Sheet

What our labs measured about tokens, prompting, RAG, evals, agents, serving, cost and fine-tuning, and the rule each result teaches.

where to go from here

Pick the next step that fits you.

self-paced course

AI Engineering course

Learn to build RAG apps, AI agents and evals step by step, with no heavy maths.

219 lessons, 10 free to read. Lifetime access.

₹1,333 once in India · $35 once elsewhere

You own this course

See the course

live classes

AI Engineering Live

Build one real AI support copilot from an empty folder, live with Roni Das, over 14 Saturdays.

Weekly on Saturdays, 8:30 PM IST. Recordings of past classes included.

₹15,000 full fee in India · $300 full fee elsewhere

See the live classes

self-paced course

System Design course

Learn how real systems are built and scaled, from your first database to a staff-level design interview.

770 lessons, 18 free to read. Lifetime access.

₹1,111 once in India · $25 once elsewhere

You own this course

See the course

interview prep

Interview prep packs

Real interview questions with a full answer and a diagram for each one, as PDFs you keep.

9 packs on sale. The one closest to this page is the Inference Engineering pack.

₹999 a pack, or all nine for ₹3,000 in India · $20 a pack, or all nine for $49 elsewhere

See the packs

Both courses together: ₹1,999 in India / $49 elsewhere, lifetime access.

System Design and AI Engineering in one payment. Bought one by one: ₹1,111 + ₹1,333 = ₹2,444, so you save ₹445. Bought one by one: $25 + $35 = $60, so you save $11.

Get both courses

Also open for enrolment: System Design Interview Bootcamp and Neo4j, GDS and GraphRAG. See all programmes

the question most readers have next

What would it take to put running and sizing LLM inference into production?

A demo works once. Production has to work every time, under load and on bad input. The lessons below cover what that takes, starting with "Quantization: The Same Model in Fewer Bits".

The lessons that teach the next part

The same Qwen2.5 3B model stored three ways and measured on a laptop: 6.18 GB at 16 bits per weight, 3.29 GB at 8.5 and 1.93 GB at about 5. The 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts and got the same 33 of 40 sums right; the 5-bit file kept 19 of 24, all 5 changes on near ties. Writing speed went from 11.1 to 18.3 to 28.4 tokens a second, while reading barely changed.
Quantization: The Same Model in Fewer BitsAI Engineering course46 min
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
The Context Window: What Happens When a Prompt Does Not FitAI Engineering course46 min
Attention lets every word in a prompt gather meaning from the words before it. Worked by hand with tiny numbers, then read from a real model, Qwen2.5-0.5B: a median of half of each word's attention went to the very first word, a word never looked ahead, and on 10 sentences where one word changes what 'it' means, the model was right 6 times and gave the same answer for both versions in 4 of 5 pairs.
Attention: How Each Word Looks Back at the Words Before ItAI Engineering course49 min
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
LLM Inference Optimization: Serving More Tokens Per GPUAI Engineering course60 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free AI Engineering cheat sheet (PDF)

Where this is taught

Generative AI and Agentic AI Engineering

219 lessons · about 230 hours · 10 free to read

₹1,333 in India by UPI$35 elsewhere by PayPal, once, for life

See the Generative AI and Agentic AI course

You own this course

Also by the same author

System Design Masterclass

770 lessons · about 290 hours · 18 free to read

₹1,111 in India by UPI$25 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Questions

What is llama.cpp 1.0?

llama.cpp 1.0.0 is the first stable release of llama.cpp, the open-source C and C++ engine for running large language models locally, released on 13 November 2026 together with ggml 1.0.0. From this version a breaking change to its public C API is marked by a new major version number.

What is new in llama.cpp 1.0.0?

It adopts semantic versioning, removes 35 deprecated functions and the --defrag-thold flag, adds an FP8 KV cache and a host-memory cache for mixture-of-experts models, supports new models such as Kimi-K3.1-Pro, adds a text-to-speech endpoint to llama-server, tightens server security and redesigns the built-in web chat.

What is the FP8 KV cache in llama.cpp?

It stores the attention keys and values in an 8-bit floating point format, f8_e4m3, instead of 16-bit, using -ctk f8_e4m3 -ctv f8_e4m3. In our test with Qwen3 0.6B and a 32,768-token context it cut the cache from 3,584 MiB to 1,792 MiB. On our Apple M4, it only started with the model on the CPU.

How do I upgrade to llama.cpp 1.0?

With Homebrew, run brew upgrade llama.cpp and check llama-server --version. Remove the --defrag-thold flag from any scripts, and if you build against the C library, rebuild and replace any of the removed deprecated functions.

Is llama.cpp the same as Ollama or the Llama app?

No. llama.cpp is the engine. Ollama is a separate project that runs models with its own tooling. Llama is a desktop app for Mac and Windows from the llama.cpp team and Hugging Face that uses llama.cpp and offers a chat window and an OpenAI-compatible API on your computer.

Does llama.cpp 1.0 work on a Mac?

Yes. It runs on Apple Silicon Macs with the Metal GPU backend, and is available through Homebrew and MacPorts. In our test the default f16 and the q8_0 KV caches worked on the GPU, but the new FP8 cache did not start there.

Sources

What this explainer is based on, so you can check it.

llama.cpp v1.0.0 release notes (13 November 2026) ggml-org on GitHub
llama.cpp v1.0.0: A stronger foundation for the local AI ecosystem Hugging Face
ggml v1.0.0 release ggml-org on GitHub
Releases and versioning of ggml-org projects ggml-org on GitHub
Llama, the desktop app llama.cpp team and Hugging Face
Homebrew formula for llama.cpp Homebrew

Was this helpful?

Learn the parts behind it

Quantization: The Same Model in Fewer BitsThe same Qwen2.5 3B model stored three ways and measured on a laptop: 6.18 GB at 16 bits per weight, 3.29 GB at 8.5 and 1.93 GB at about 5. The 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts and got the same 33 of 40 sums right; the 5-bit file kept 19 of 24, all 5 changes on near ties. Writing speed went from 11.1 to 18.3 to 28.4 tokens a second, while reading barely changed.46 min lesson The Context Window: What Happens When a Prompt Does Not FitMeasured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.46 min lesson Attention: How Each Word Looks Back at the Words Before ItAttention lets every word in a prompt gather meaning from the words before it. Worked by hand with tiny numbers, then read from a real model, Qwen2.5-0.5B: a median of half of each word's attention went to the very first word, a word never looked ahead, and on 10 sentences where one word changes what 'it' means, the model was right 6 times and gave the same answer for both versions in 4 of 5 pairs.49 min lesson LLM Inference Optimization: Serving More Tokens Per GPUWhy LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding60 min lesson One Box, Many Models: How Many Fit, and What Happens When They Don't?I put many copies of a model on one small rented computer (1 core, 1.9 GB of memory) and measured how much each took, how many fit, and what happened when one more did not. A 14.3 MB model file took 15.1 MB of memory once loaded, but Python and its tool libraries took 198 MB before any model. Under a 537 MB limit, 27 copies fit. The computer then stopped the program while it loaded the 28th, with no error message. Four copies of a service that loaded the models once and then copied themselves used 385 MB, against 1,080 MB when each loaded its own. One clean-up by Python's memory cleaner copied 45 MB more per copy, and one setting (gc.freeze) stopped that. When only 4 of 16 models fit and the disk was slow, even requests whose model was ready waited up to 87 ms behind loads. Loading in the background cut that to 6 ms. Every step on the rented computer is recorded, so you can build it yourself.95 min lesson

Go deeper than the news

This explainer sits on top of our AI Engineering course: 219 lessons on RAG, evals, agents, serving, security and MLOps, many built around a real experiment. 10 lessons are free to read, with no card needed.

course 2

Generative AI and Agentic AI Engineering

Build RAG apps, AI agents and LLM systems. No heavy maths, taught from the basics, with labs on real models.

219 interactive lessons
RAG, AI agents, evals and fine-tuning
Serving, pipelines, feature stores and monitoring
Step-by-step diagrams and quizzes
Lifetime access and all future lessons
Sold on its own, or with System Design in one payment

Read 10 lessons free first

In India

₹1,333

Pay with UPI

Everywhere else

$35

Pay by card or PayPal

once · yours for life

You own this course

Continue with Quantization: The Same Model in Fewer Bits

Explainedllama.cppggmllocal LLMLLM inferenceKV cachequantizationHugging Face

llama.cpp 1.0 explained: what the first stable version changes, the new FP8 KV cache, and how to upgrade

This explainer covers one piece of news. The AI Engineering course teaches how systems like this work, step by step.

This page's course

The AI Engineering course

₹1,333 / $35 once, with lifetime access.

Start the first free lesson See the course

Best value

Both courses together

₹1,999 / $49 once, for System Design and AI Engineering, with lifetime access to both.

Get both courses

More ways to learn

All 9 interview packs: ₹3,000 / $49, every question with a full answer and a diagram, one payment.
System Design Interview Bootcamp: ₹15,000 / $300, self-paced, every class recorded.
AI Engineering Live: ₹15,000 / $300, live Saturday classes, and every class recorded.

Free PDF: the AI Engineering cheat sheet, 12 pages Get it free

Roni Das

8 min read

Published14 November 2026, 04:15 UTCUpdated14 November 2026, 04:15 UTC

Two courses by the author of this page:

Generative AI and Agentic AI

219 lessons · 10 free to read

₹1,333 in India · $35 elsewhere, once

Get Generative AI and Agentic AI

You own this course

System Design

770 lessons · 18 free to read

₹1,111 in India · $25 elsewhere, once

Get System Design

You own this course

Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor

In short

llama.cpp 1.0.0 and ggml 1.0.0 were released on 13 November 2026 by the ggml-org team at Hugging Face, about three and a half years after the project began.
From now on a breaking change to the public C API means a new major version, a new feature or model a minor version, and a bug fix a patch version.
Version 1.0.0 removes 35 deprecated functions and the --defrag-thold flag, so old code and scripts can break when you upgrade.
It adds an FP8 KV cache (-ctk f8_e4m3 -ctv f8_e4m3). In our run it halved the cache, 3,584 MiB to 1,792 MiB, but would not start on the Apple GPU.
llama-server gets safer defaults: tools and private media URLs now need explicit opt-in flags, and CORS, DNS rebinding and SSRF problems were fixed.

Released: 13 November 2026
Version: llama.cpp 1.0.0, ggml 1.0.0
Previous: 0.9.0
Model architectures: more than 150
License: open source, MIT

This page is the free part.

The course goes deeper on running and sizing LLM inference

₹1,333 in India$35 everywhere elseonce, for the whole course

Quantization: The Same Model in Fewer Bits
The same Qwen2.5 3B model stored three ways and measured on a laptop: 6.18 GB at 16 bits per weight, 3.29 GB at 8.5 and 1.93 GB at about 5. The 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts and got the same 33 of 40 sums right; the 5-bit file kept 19 of 24, all 5 changes on near ties. Writing speed went from 11.1 to 18.3 to 28.4 tokens a second, while reading barely changed.
46 min · diagrams · code you run · quiz
The Context Window: What Happens When a Prompt Does Not Fit
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
46 min · diagrams · code you run · quiz
Attention: How Each Word Looks Back at the Words Before It
Attention lets every word in a prompt gather meaning from the words before it. Worked by hand with tiny numbers, then read from a real model, Qwen2.5-0.5B: a median of half of each word's attention went to the very first word, a word never looked ahead, and on 10 sentences where one word changes what 'it' means, the model was right 6 times and gave the same answer for both versions in 4 of 5 pairs.
49 min · diagrams · code you run · quiz
LLM Inference Optimization: Serving More Tokens Per GPU
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
60 min · diagrams · code you run · quiz

all part of

Generative AI and Agentic AI Engineering

219 lessons · about 230 hours · 10 free to read

₹1,333in India, by UPI

$35everywhere else, by PayPal

Get the Generative AI and Agentic AI course Read 10 lessons free first

You own this course

Continue with Quantization: The Same Model in Fewer Bits

pay once,
yours for life

What is llama.cpp 1.0?

“Moving forward, both llama.cpp and ggml will reflect API and ABI incompatibilities with major-version bumps.”

ggml-org, llama.cpp v1.0.0 on the Hugging Face blog, 13 November 2026

What do the version numbers mean now?

Plans and limits

Major, for example 1.x to 2.0: a breaking change to the public C API in ./include
Minor, for example 1.0 to 1.1: a new feature, backend update, model or API addition that does not break anything
Patch, for example 1.0.0 to 1.0.1: a bug fix with no API change

What else is new in llama.cpp 1.0.0?

The release notes are long. These are the changes most people will notice.

FP8 KV cache. A new cache type, f8_e4m3, stores the attention keys and values in one byte each instead of two. Turn it on with -ctk f8_e4m3 -ctv f8_e4m3.
MoE expert cache. For mixture-of-experts models, a cache in host memory for experts that do not fit on the GPU, with a setting for how much of the missed work the CPU does.
Safer llama-server. Tools without authentication and fetching media from private addresses now need the --allow-unauthenticated-tools and --media-allow-private-urls flags. CORS origins, DNS rebinding and a server-side request forgery hole in the CORS proxy were fixed, and the HF_TOKEN is only sent to huggingface.co.
New models. Kimi-K3.1-Pro, Ripple-65B-Preview, xing4_0 and the drama-3 text-to-speech model, plus MageVL-Cowork and Kimi-K3.1 vision.
Speech endpoint. llama-server adds an OpenAI-compatible /v1/audio/speech endpoint for text-to-speech models.
New web interface. The built-in chat page was redesigned, with chat groups, a markdown editor and better layouts on phones.

everything we teach

Two courses, interview packs and live classes, all by the author of this page.

self-paced course

AI Engineering course

Learn to build RAG apps, AI agents and evals step by step, with no heavy maths.

219 lessons, 10 free to read. Lifetime access.

₹1,333 once in India · $35 once elsewhere

You own this course

See the course

live classes

AI Engineering Live

Build one real AI support copilot from an empty folder, live with Roni Das, over 14 Saturdays.

Weekly on Saturdays, 8:30 PM IST. Recordings of past classes included.

₹15,000 full fee in India · $300 full fee elsewhere

See the live classes

self-paced course

System Design course

Learn how real systems are built and scaled, from your first database to a staff-level design interview.

770 lessons, 18 free to read. Lifetime access.

₹1,111 once in India · $25 once elsewhere

You own this course

See the course

interview prep

Interview prep packs

Real interview questions with a full answer and a diagram for each one, as PDFs you keep.

9 packs on sale. The one closest to this page is the Inference Engineering pack.

₹999 a pack, or all nine for ₹3,000 in India · $20 a pack, or all nine for $49 elsewhere

See the packs

Both courses together: ₹1,999 in India / $49 elsewhere, lifetime access.

System Design and AI Engineering in one payment. Bought one by one: ₹1,111 + ₹1,333 = ₹2,444, so you save ₹445. Bought one by one: $25 + $35 = $60, so you save $11.

Get both courses

Also open for enrolment: System Design Interview Bootcamp and Neo4j, GDS and GraphRAG. See all programmes

Hands-on: the FP8 KV cache, the new server checks and a removed flag

$ MODEL=Qwen3-0.6B-Q8_0.gguf bash kv_cache_check.sh
version: 1.0.0 (build 12359, commit b1b770d90)

1. KV cache size by type (model on the CPU, -ngl 0)
f16      3584.00 MiB
f8_e4m3  1792.00 MiB
q8_0     1904.00 MiB

3. CORS preflight with --cors-origins http://localhost:3000
from http://localhost:3000 -> Access-Control-Allow-Origin: http://localhost:3000
from https://evil.example -> Access-Control-Allow-Origin: http://localhost:3000

4. The same caches with the model on the Apple GPU (Metal, the default on a Mac)
f16      started, 3584.00 MiB
f8_e4m3  server exited: pre-allocated tensor (cache_k_l0 (view)) in a buffer (MTL0) that cannot run the operation (SET_ROWS)
q8_0     started, 1904.00 MiB

5. A flag that 1.0.0 removed
llama-server --defrag-thold 0.1 -> error: invalid argument: --defrag-thold

if this is what you came for

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

start here

Generative AI and Agentic AI Engineering

How models generate text: tokens, attention, the context window, the KV cache, quantization and serving, each explained from first principles with diagrams and a quiz.

219 lessons in 17 chapters: RAG, AI agents, evals, fine-tuning and serving
No heavy maths, everything taught step by step from the basics
127 lessons built on a lab that really ran
An AI tutor beside every lesson that answers and draws
Lifetime access, every new lesson included

₹1,333 once, by UPI

$35 once, by PayPal

$35 outside India

See the Generative AI and Agentic AI course Buy Generative AI and Agentic AI

You own this course Open it

Not sure yet? 10 lessons are free to read, no card needed.

How to upgrade to llama.cpp 1.0 without breaking anything

For most people who use the command line tools or an app built on llama.cpp, the upgrade is quiet. If you call the C library yourself or pass old flags in scripts, check these first.

Upgrade
On a Mac or Linux with Homebrew, run brew upgrade llama.cpp. Check the result with llama-server --version, which should print version 1.0.0.
Search your scripts for removed flags
--defrag-thold is gone and now stops llama-server with an invalid argument error. Remove it from start-up scripts and service files.
Rebuild against the new headers
35 deprecated functions and llama_set_warmup were removed from llama.h, and llama_set_embd_state_src is now llama_set_ctx_state_src. If your code still compiles, it does not use them.
Multimodal code
clip.h was removed and its work moved into the mtmd library. Code that included clip.h directly must move to mtmd.
Test a smaller KV cache
Try -ctk f8_e4m3 -ctv f8_e4m3 or q8_0 on your own hardware and model, compare the answers on your own prompts, and keep whichever works and stays accurate enough.
Lock down the server
If anything other than your own machine can reach llama-server, set --api-key and --cors-origins, and only add the new opt-in flags if you need them.

What does llama.cpp 1.0 mean for engineers?

if this explainer was useful

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

start here

Generative AI and Agentic AI Engineering

How models generate text: tokens, attention, the context window, the KV cache, quantization and serving, each explained from first principles with diagrams and a quiz.

219 lessons in 17 chapters: RAG, AI agents, evals, fine-tuning and serving
No heavy maths, everything taught step by step from the basics
127 lessons built on a lab that really ran
An AI tutor beside every lesson that answers and draws
Lifetime access, every new lesson included

₹1,333 once, by UPI

$35 once, by PayPal

$35 outside India

See the Generative AI and Agentic AI course Buy Generative AI and Agentic AI

You own this course Open it

Not sure yet? 10 lessons are free to read, no card needed.

Free PDF · 12 pages

Get the free AI Engineering Cheat Sheet

What our labs measured about tokens, prompting, RAG, evals, agents, serving, cost and fine-tuning, and the rule each result teaches.

where to go from here

Pick the next step that fits you.

self-paced course

AI Engineering course

Learn to build RAG apps, AI agents and evals step by step, with no heavy maths.

219 lessons, 10 free to read. Lifetime access.

₹1,333 once in India · $35 once elsewhere

You own this course

See the course

live classes

AI Engineering Live

Build one real AI support copilot from an empty folder, live with Roni Das, over 14 Saturdays.

Weekly on Saturdays, 8:30 PM IST. Recordings of past classes included.

₹15,000 full fee in India · $300 full fee elsewhere

See the live classes

self-paced course

System Design course

Learn how real systems are built and scaled, from your first database to a staff-level design interview.

770 lessons, 18 free to read. Lifetime access.

₹1,111 once in India · $25 once elsewhere

You own this course

See the course

interview prep

Interview prep packs

Real interview questions with a full answer and a diagram for each one, as PDFs you keep.

9 packs on sale. The one closest to this page is the Inference Engineering pack.

₹999 a pack, or all nine for ₹3,000 in India · $20 a pack, or all nine for $49 elsewhere

See the packs

Both courses together: ₹1,999 in India / $49 elsewhere, lifetime access.

System Design and AI Engineering in one payment. Bought one by one: ₹1,111 + ₹1,333 = ₹2,444, so you save ₹445. Bought one by one: $25 + $35 = $60, so you save $11.

Get both courses

Also open for enrolment: System Design Interview Bootcamp and Neo4j, GDS and GraphRAG. See all programmes

the question most readers have next

What would it take to put running and sizing LLM inference into production?

A demo works once. Production has to work every time, under load and on bad input. The lessons below cover what that takes, starting with "Quantization: The Same Model in Fewer Bits".

The lessons that teach the next part

The same Qwen2.5 3B model stored three ways and measured on a laptop: 6.18 GB at 16 bits per weight, 3.29 GB at 8.5 and 1.93 GB at about 5. The 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts and got the same 33 of 40 sums right; the 5-bit file kept 19 of 24, all 5 changes on near ties. Writing speed went from 11.1 to 18.3 to 28.4 tokens a second, while reading barely changed.
Quantization: The Same Model in Fewer BitsAI Engineering course46 min
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
The Context Window: What Happens When a Prompt Does Not FitAI Engineering course46 min
Attention lets every word in a prompt gather meaning from the words before it. Worked by hand with tiny numbers, then read from a real model, Qwen2.5-0.5B: a median of half of each word's attention went to the very first word, a word never looked ahead, and on 10 sentences where one word changes what 'it' means, the model was right 6 times and gave the same answer for both versions in 4 of 5 pairs.
Attention: How Each Word Looks Back at the Words Before ItAI Engineering course49 min
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
LLM Inference Optimization: Serving More Tokens Per GPUAI Engineering course60 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free AI Engineering cheat sheet (PDF)

Where this is taught

Generative AI and Agentic AI Engineering

219 lessons · about 230 hours · 10 free to read

₹1,333 in India by UPI$35 elsewhere by PayPal, once, for life

See the Generative AI and Agentic AI course

You own this course

Also by the same author

System Design Masterclass

770 lessons · about 290 hours · 18 free to read

₹1,111 in India by UPI$25 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Questions

What is llama.cpp 1.0?

What is new in llama.cpp 1.0.0?

What is the FP8 KV cache in llama.cpp?

How do I upgrade to llama.cpp 1.0?

Is llama.cpp the same as Ollama or the Llama app?

Does llama.cpp 1.0 work on a Mac?

Sources

What this explainer is based on, so you can check it.

llama.cpp v1.0.0 release notes (13 November 2026) ggml-org on GitHub
llama.cpp v1.0.0: A stronger foundation for the local AI ecosystem Hugging Face
ggml v1.0.0 release ggml-org on GitHub
Releases and versioning of ggml-org projects ggml-org on GitHub
Llama, the desktop app llama.cpp team and Hugging Face
Homebrew formula for llama.cpp Homebrew

Was this helpful?

Learn the parts behind it

Go deeper than the news

course 2

Generative AI and Agentic AI Engineering

Build RAG apps, AI agents and LLM systems. No heavy maths, taught from the basics, with labs on real models.

219 interactive lessons
RAG, AI agents, evals and fine-tuning
Serving, pipelines, feature stores and monitoring
Step-by-step diagrams and quizzes
Lifetime access and all future lessons
Sold on its own, or with System Design in one payment

Read 10 lessons free first

In India

₹1,333

Pay with UPI

Everywhere else

$35

Pay by card or PayPal

once · yours for life

You own this course

Continue with Quantization: The Same Model in Fewer Bits

llama.cpp 1.0 explained: what the first stable version changes, the new FP8 KV cache, and how to upgrade

In short

The course goes deeper on running and sizing LLM inference

What is llama.cpp 1.0?

What do the version numbers mean now?

What else is new in llama.cpp 1.0.0?

Two courses, interview packs and live classes, all by the author of this page.

Hands-on: the FP8 KV cache, the new server checks and a removed flag

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

How to upgrade to llama.cpp 1.0 without breaking anything

Upgrade

Search your scripts for removed flags

Rebuild against the new headers

Multimodal code

Test a smaller KV cache

Lock down the server

What does llama.cpp 1.0 mean for engineers?

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

Get the free AI Engineering Cheat Sheet

Pick the next step that fits you.

What would it take to put running and sizing LLM inference into production?

The lessons that teach the next part

Questions

What is llama.cpp 1.0?

What is new in llama.cpp 1.0.0?

What is the FP8 KV cache in llama.cpp?

How do I upgrade to llama.cpp 1.0?

Is llama.cpp the same as Ollama or the Llama app?

Does llama.cpp 1.0 work on a Mac?

Sources

Learn the parts behind it

Go deeper than the news

Generative AI and Agentic AI Engineering

Get all 120 questions in the Inference Engineering pack

llama.cpp 1.0 explained: what the first stable version changes, the new FP8 KV cache, and how to upgrade

In short

The course goes deeper on running and sizing LLM inference

What is llama.cpp 1.0?

What do the version numbers mean now?

What else is new in llama.cpp 1.0.0?

Two courses, interview packs and live classes, all by the author of this page.

Hands-on: the FP8 KV cache, the new server checks and a removed flag

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

How to upgrade to llama.cpp 1.0 without breaking anything

Upgrade

Search your scripts for removed flags

Rebuild against the new headers

Multimodal code

Test a smaller KV cache

Lock down the server

What does llama.cpp 1.0 mean for engineers?

Want to understand what the KV cache is, why it eats memory, and how quantization shrinks a model? Start with Generative AI and Agentic AI.

Get the free AI Engineering Cheat Sheet

Pick the next step that fits you.

What would it take to put running and sizing LLM inference into production?

The lessons that teach the next part

Questions

What is llama.cpp 1.0?

What is new in llama.cpp 1.0.0?

What is the FP8 KV cache in llama.cpp?

How do I upgrade to llama.cpp 1.0?

Is llama.cpp the same as Ollama or the Llama app?

Does llama.cpp 1.0 work on a Mac?

Sources

Learn the parts behind it

Go deeper than the news

Generative AI and Agentic AI Engineering

Get all 120 questions in the Inference Engineering pack