free exercise · worked answer · real lab output
What does 1 million tokens cost to self-host on RunPod compared with an API?
Our measured RTX 4090 speed, dated RunPod and API prices, and the one number that decides it: how busy the GPU is.
The scenario
A small startup sends about 2 million short questions a month to a support bot. Each answer is around 300 tokens. They pay an API by the token today, and an engineer has proposed renting an RTX 4090 on RunPod and running an open 8B model on it instead. He says it will be "basically free".
On 13 November 2026 RunPod listed the RTX 4090 at $0.34 an hour on its community cloud and $0.89 an hour on its secure cloud. Together AI listed Qwen3.5 9B, a model of a similar size, at $0.17 per million input tokens and $0.25 per million output tokens. We measured Qwen3-8B in 4-bit on a RunPod RTX 4090 at 164 tokens a second for one person. The two models are not the same: Together serves a different, newer model, Qwen3.5 9B, and a 4-bit file gives up a little quality, so check that the answers are good enough before you compare prices.
Your task: work out what a million output tokens cost on the rented RTX 4090, for one person at a time, with several people at once, and when the GPU is busy only part of the day. Then say who is right.
Words this exercise uses
- Output and input tokens
- Input tokens are the prompt you send; output tokens are the answer the model writes. APIs price them separately.
- Community and secure cloud
- RunPod's community cloud runs on machines other people host and is cheaper; its secure cloud runs in vetted data centres and costs more.
- Batching
- Letting several conversations share the GPU at the same moment, so one read of the weights serves all of them.
- Busy share
- The part of the hours you pay for in which the GPU actually has work.
Show a hint
You pay for the GPU by the hour whether it writes or waits. So the cost of a million tokens is the hourly price divided by the tokens written in that hour, times one million.
Two things change the tokens in an hour a lot: how many conversations share the GPU at once, and what share of the hour it has any work at all.
Show the worked answer
Step 1. One person at a time
cost of a million = $0.34 / 590,364 x 1,000,000 = $0.58 (community cloud)
on the secure cloud = $0.89 / 590,364 x 1,000,000 = $1.51
That is more than twice the API's $0.25 for output, before counting anything else. To match $0.25, the card would have to write $0.34 / ($0.25 / 1,000,000) / 3,600 = 378 tokens a second, all hour.
Step 2. Several people at once
Writing a token is limited by reading the weights from memory, and one read can serve every conversation in the batch. We measured how much that helps on the same RTX 4090 with a small model, Qwen3-0.6B, using llama.cpp's batched benchmark (S_TG is the total tokens written per second):
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 256 | 128 | 1 | 384 | 0.033 | 7713.17 | 0.218 | 586.10 | 0.252 | 1526.35 | | 256 | 128 | 4 | 1536 | 0.019 | 53026.77 | 0.289 | 1772.08 | 0.308 | 4983.18 | | 256 | 128 | 8 | 3072 | 0.038 | 53591.52 | 0.395 | 2589.63 | 0.434 | 7084.23 | | 256 | 128 | 32 | 12288 | 0.127 | 64543.07 | 0.844 | 4853.91 | 0.971 | 12657.88 |
Our RTX 4090, 13 November 2026: llama-batched-bench, Qwen3-0.6B Q8_0, 256-token prompts and 128-token answers.
With 8 at once, the card wrote 4.4 times as many tokens in total as with one. We did not run the batched benchmark with the 8B model, so for it we use a roofline estimate (the best-case speed from memory bandwidth, explained in the concurrent users exercise): 881 tokens a second in total with 8 at once. It estimates 132 for one person, a little under the 164 we measured, but its gain from batching, 6.7 times, is larger than the 4.4 times we measured on the small model, so treat the next numbers as hopeful.
Step 3. The GPU is not busy all day
A support bot is quiet at night and at weekends, and a rented GPU costs the same per hour whether it works or waits. If it has work for only 30% of the hours you pay for:
| Setting | Tokens a second | Busy | Cost per million |
|---|---|---|---|
| One at a time, community cloud (measured speed) | 164 | 100% | $0.58 |
| One at a time, secure cloud (measured speed) | 164 | 100% | $1.51 |
| 8 at once, community cloud (estimated speed) | 881 | 100% | $0.107 |
| 8 at once, community cloud (estimated speed) | 881 | 30% | $0.36 |
| Together AI, Qwen3.5 9B output (list price) | $0.25 |
Step 4. Do not forget the prompt
The API also charges $0.17 for every million prompt tokens. On your own GPU, reading the prompt is fast: our card read prompts at 12,095 tokens a second.
Step 5. Put the startup's month together
input, if each question with its context is about 500 tokens: 1,000 million tokens; at $0.17 = $170.00
API total = about $320.00 a month
RTX 4090 left running all month: 730 hours x $0.34 = $248.20
busy hours it needs: about 189 writing answers at 881 tokens a second, plus about 23 reading prompts at 12,095 tokens a second = about 212 of the 730
The 500-token prompt is our assumption for this example; put in your own. At this traffic the rented card comes out cheaper on paper, $248.20 against about $320.00, because there is enough steady traffic to keep several conversations on it at once. But the card has to be kept running, updated and watched, and that time is not in the sum. One RTX 4090 is also one point of failure: if the pod is reclaimed, the bot stops.
Step 6. Try your own numbers
The calculator is set to Qwen3-8B on RunPod's community RTX 4090 with our measured 164 tokens a second and the API price of $0.25. Change the busy share, or clear the measured speed to 0 to use the estimate for more people at once. The full tool is the free LLM cost calculator.
What does a million tokens cost on your own GPU?
cost per million tokensYou pay for a GPU by the hour, whether it is busy or not. Divide that price by the tokens it writes in an hour and you get the price of a million tokens, which you can set against what an API charges.
Cost of one million tokens
$0.576
at 100% busy
Tokens written per second
164
your measured number
GPUs you pay for
1
$0.340 an hour
Tokens in one paid hour
0.6 million
Your GPU against the API
Your RTX 4090: $0.576 a million tokens
The API you typed in: $0.250 a million tokens
At these settings the API is 2.3 times cheaper per token. To match the API's price, the GPU must write 378 tokens a second at 100% busy. Your own GPU also costs your time to run it, which this does not count.
Cost per million tokens = price per hour x GPUs / (tokens a second x 3,600 x busy share) x 1,000,000
= $0.34 x 1 / (164 x 3,600 x 1.00) x 1,000,000 = $0.576
The assumptions behind these numbers
- The speed is the roofline estimate for a 1,024-token prompt and a 256-token answer at 85% of memory speed and 50% of compute speed. A measured number from a benchmark is always better; type it in above.
- It counts the tokens the model writes. Reading the prompt also takes GPU time, so a service with very long prompts and short answers costs more per written token than this shows.
- Prices are on-demand list prices for one GPU, read on 13 November 2026 from RunPod's public price list, AWS's us-east-1 price file and Google Cloud's us-central1 price page. Cloud machines include their CPUs and memory. Prices change often.
- Storage, network and the engineer who keeps it running are not counted.
The GPUs in this list: NVIDIA L4, RTX 4090, A10G (A10 specs), NVIDIA L40S, A100 40 GB, A100 80 GB SXM, H100 80 GB SXM, H200 141 GB SXM.
The answer. One person at a time, the rented RTX 4090 costs $0.58 a million tokens (measured speed), more than twice the API. With 8 conversations sharing the card our estimate falls to about $0.11, below the API, but only while the card is busy; at 30% busy it is about $0.36. "Basically free" is wrong. Self-hosting pays when you keep the GPU full, and an API wins for quiet or bursty traffic.
How many people one card can hold is worked out in the concurrent users exercise.
Questions people ask
How do I work out the cost per million tokens of my own GPU?
Divide the price of the GPU for one hour by the tokens it writes in that hour, then multiply by one million. At $0.34 an hour and 164 tokens a second, one hour writes 590,400 tokens, so a million tokens cost $0.58.
Is self-hosting an LLM cheaper than an API?
Only when the GPU is busy most of the time. With one user at a time our RTX 4090 cost $0.58 per million tokens, more than twice the $0.25 that Together AI charged for a similar small model. With several users at once our estimate falls to about $0.10, but if the GPU is busy only 30% of the hour it rises to about $0.34 again.
Why does batching lower the cost?
Writing a token is limited by how fast the GPU reads its weights, and one read can serve many conversations at the same step. On our RTX 4090 a small model wrote 4.4 times more tokens in total with 8 people at once than with one.
Does the prompt cost anything when I self-host?
Very little next to the answer. Our RTX 4090 read prompts at about 12,000 tokens a second, so a million prompt tokens took about 83 seconds of GPU time, under one cent at $0.34 an hour.
What does the per-token price leave out?
Your time to run and update the server, storage, network, and the hours the GPU is rented but idle. An API also gives you scaling and uptime without work. Count those before you decide.
Where every number comes from
- Our RTX 4090 on RunPod, 13 Nov 2026: nvidia-smi 24564 MiB; llama-bench Qwen3-8B Q4_K_M 163.99 tokens a second (tg128) and 12,095 (pp512), 114.90 at a depth of 16,384; vLLM Qwen3-8B BF16 with --max-model-len 32768: model loading 15.27 GiB, KV cache 4.56 GiB, 33,216 tokens
- RunPod GPU price list: RTX 4090 at $0.34 an hour (community cloud) and $0.89 (secure cloud), L40S $0.79 and $1.09, H100 $2.69 and $4.39, read 13 Nov 2026
- Together AI serverless prices: Qwen3.5 9B $0.17 per million input tokens and $0.25 per million output tokens; Qwen2.5 7B Instruct Turbo $0.30 and $0.30, read 13 Nov 2026
- Our own run on an Apple M4 MacBook Air with 24 GB, 12 Nov 2026: recommendedMaxWorkingSetSize = 17179.89 MB, and Qwen3-8B Q4_K_M at 20.5 tokens a second (llama.cpp build 12110)
This exercise comes from the course Hands-On Inference Engineering, where each step is done on a real GPU and explained line by line. More exercises like it: Which LLMs fit on a Mac; Which LLMs fit on an RTX 4090; Users per GPU on an L40S or H100; Tensor vs pipeline parallelism for 70B. Or see every free exercise.