| decode | NVIDIA GeForce RTX 3080 (10 GB) | 34.40 tokens a second | decode, tokens a second, split with system memory | gemma4:26b | not stated | not stated | not stated | A Short History of Mistral, § What we measured on two cards in one box |
| decode | NVIDIA GeForce RTX 3080 (10 GB) | 5.13 | decode, tokens a second, split with system memory | mistral-small3.2:24b | not stated | not stated | not stated | A Short History of Mistral, § What we measured on two cards in one box |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 115.4 | decode, tokens a second | gemma4:26b | Ollama | 0.32.13 | 300 W each, 600 W for both cards | Two 3080s against one 3090, and the cap decides, § How fast does each one write? |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 113.8 tok/s, 18,815 MiB | decode at the full window, and the memory it used | gemma4:26b | Ollama | 0.32.13 | not stated | Two 3080s against one 3090, and the cap decides, § What the cache type costs |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 114.8 | decode, tokens a second | gemma4:26b | Ollama | 0.32.13 | 250 W each, 500 W for both cards | Two 3080s against one 3090, and the cap decides, § How fast does each one write? |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 124.0 | decode summed across conversations, tokens a second | gemma4:26b | Ollama | 0.32.13 | 300 W each, 600 W for both cards | Two 3080s against one 3090, and the cap decides, § The full concurrency ladder |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 124.4 | decode summed across conversations, tokens a second | gemma4:26b | Ollama | 0.32.13 | 250 W each, 500 W for both cards | Two 3080s against one 3090, and the cap decides, § The full concurrency ladder |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 124.7 | decode out of the box once the layer count is named | gemma4:26b | Ollama | 0.32.13 | not stated | Two 3080s against one 3090, and the cap decides, § At a glance |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 44.6 | decode, tokens a second | mistral-small3.2:24b | Ollama | 0.32.13 | 250 W each, 500 W for both cards | Two 3080s against one 3090, and the cap decides, § How fast does each one write? |
| decode | Two NVIDIA GeForce RTX 3080 (10 GB each) | 44.7 | decode, tokens a second | mistral-small3.2:24b | Ollama | 0.32.13 | 300 W each, 600 W for both cards | Two 3080s against one 3090, and the cap decides, § How fast does each one write? |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 85.007 | decode per request, tokens a second | Gemma 4 12B | Ollama | 0.34.4 | 300 W | Ollama and vLLM on one RTX 3090, § Requests at once |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 441.866 | the wait for the first token, 95th percentile, ms | Gemma 4 12B | Ollama | 0.34.4 | 300 W | Ollama and vLLM on one RTX 3090, § Requests at once |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 82.693 | decode per request, tokens a second | Gemma 4 12B | vLLM | 0.30.0 | 300 W | Ollama and vLLM on one RTX 3090, § Requests at once |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 212.999 | the wait for the first token, 95th percentile, ms | Gemma 4 12B | vLLM | 0.30.0 | 300 W | Ollama and vLLM on one RTX 3090, § Requests at once |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 136.274 | decode summed across conversations, tokens a second | gemma4:26b | Ollama | 0.32.13 | 350 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § Three more arms |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 135.249, 19,415 MiB | decode at the full window, and the memory it used | gemma4:26b | Ollama | 0.32.13 | 350 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § What fits, and what the cache costs |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 136.618 | decode, tokens a second | gemma4:26b | Ollama | 0.32.13 | 350 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § What a power cap takes, and what each card does at its own limit |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 146.308 tok/s, 20,995 MiB | decode at the full window, and the memory it used | gemma4:26b | Ollama | 0.32.13 | 350 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § What fits, and what the cache costs |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 74.894 (0.398) | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 60.41 | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 60.545 (0.033) | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 64.83 | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 65.724 (0.441) | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 74.135 | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 78.158 | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 78.942 (0.068) | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 87.234 | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 88.203 (0.252) | writing speed, tokens a second | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 510.56 | first token, ms | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 481.8 | first token, ms | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 516.45 | first token, ms | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 518.25 | first token, ms | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| wait | NVIDIA GeForce RTX 3090 (24 GB) | 548.22 | first token, ms | mistral-nemo:12b-instruct-2407 | Ollama | 0.32.13 | 300 W | The quantization ladder, § What each step down bought: speed and memory |
| decode | NVIDIA GeForce RTX 3090 (24 GB) | 46.880 | decode, tokens a second | mistral-small3.2:24b | Ollama | 0.32.13 | 350 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § What a power cap takes, and what each card does at its own limit |
| decode | NVIDIA GeForce RTX 3090 Ti (24 GB) | 147.364 | decode summed across conversations, tokens a second | gemma4:26b | Ollama | 0.32.13 | 450 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § Three more arms |
| decode | NVIDIA GeForce RTX 3090 Ti (24 GB) | 147.560 | decode, tokens a second | gemma4:26b | Ollama | 0.32.13 | 450 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § What a power cap takes, and what each card does at its own limit |
| decode | NVIDIA GeForce RTX 3090 Ti (24 GB) | 57.396 | decode, tokens a second | mistral-small3.2:24b | Ollama | 0.32.13 | 450 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § What a power cap takes, and what each card does at its own limit |
| decode | NVIDIA GeForce RTX 3090 Ti (24 GB) | 22.657 | writing speed, tokens a second | Qwen3.8-Flash-Next | llama.cpp | b11514 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The 24 GB desktop card: both engines in one window |
| prefill | NVIDIA GeForce RTX 3090 Ti (24 GB) | 255.2 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | llama.cpp | b11514 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The 24 GB desktop card: both engines in one window |
| decode | NVIDIA GeForce RTX 3090 Ti (24 GB) | 98.291 | writing speed, tokens a second | Qwen3.8-Flash-Next | the Strata engine | 0.1.39 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The 24 GB desktop card: both engines in one window |
| prefill | NVIDIA GeForce RTX 3090 Ti (24 GB) | 2,143.1 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | the Strata engine | 0.1.39 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The 24 GB desktop card: both engines in one window |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 118.368 | summed tokens/s | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 116.125 (115.846–116.716) | tokens/s (spread of three) | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 118.424 | tokens/s | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 121.379 (120.822–123.017) | tokens/s (spread of three) | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 345.52 | first token, ms | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 326.27 | first token, ms | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 372.49 | first token, ms | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 435.96 | first token, ms | gemma4:26b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 86.808 (85.795–87.364) | tokens/s (spread of three) | minicpm-v4.5 | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 149.86 | first token, ms | minicpm-v4.5 | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 28.839 | tokens/s | mistral-small3.2:24b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 28.851 (28.721–28.993) | tokens/s (spread of three) | mistral-small3.2:24b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 210.94 | first token, ms | mistral-small3.2:24b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| wait | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 214.37 | first token, ms | mistral-small3.2:24b | Ollama | 0.32.13 | 95 W | A laptop, asked the desktop's questions, § The tables |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 22.5 | writing speed, tokens a second | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § Writing the answer (the laptop, the IQ3_S file) |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 24.4 | writing speed, tokens a second | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § Writing the answer (the laptop, the IQ3_S file) |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 32.5 | writing speed, tokens a second | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The Q2_0 pair (the laptop) |
| prefill | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 533.7 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The first word (the laptop, the IQ3_S file) |
| prefill | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 211.7 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The first word (the laptop, the IQ3_S file) |
| decode | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 85.6 | writing speed, tokens a second | Qwen3.8-Flash-Next | the Strata engine | 0.1.25 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § Writing the answer (the laptop, the IQ3_S file) |
| prefill | NVIDIA GeForce RTX 5090 Laptop GPU (24 GB) | 2,147.5 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | the Strata engine | 0.1.25 | 135–150 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The first word (the laptop, the IQ3_S file) |
| decode | NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) | 205.85 tokens a second | decode, tokens a second | gemma4:26b | Ollama | not stated | 420 W | A Short History of Mistral, § What we measured on two cards in one box |
| decode | NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) | 92.77 tokens a second | decode, tokens a second | mistral-small3.2:24b | Ollama | not stated | 420 W | A Short History of Mistral, § What we measured on two cards in one box |
| decode | NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) | 102.889 | writing speed, tokens a second | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The desktop card: one engine a night |
| prefill | NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) | 1,619.4 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | llama.cpp | 3cf03257 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The desktop card: one engine a night |
| decode | NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) | 205.263 | writing speed, tokens a second | Qwen3.8-Flash-Next | the Strata engine | 0.1.39 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The desktop card: one engine a night |
| prefill | NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) | 3,920.0 | prompt rate, tokens a second, the engine's clock | Qwen3.8-Flash-Next | the Strata engine | 0.1.39 | 300 W | Why the first word comes sooner: the Strata engine and llama.cpp on Qwen3.8-Flash-Next, the full bench, § The desktop card: one engine a night |
| prefill | NVIDIA GeForce RTX 3090 (24 GB) | 2,270.547 | prompt reading, tokens a second | not stated | Ollama | 0.32.13 | 350 W | RTX 3090 vs RTX 3090 Ti: an inference and diffusion bench, § Three more arms |