Skip to content

Sizing

Which machine do you need?

Capacity decides whether a model loads. Bandwidth decides whether it is usable. Sizing on the first and ignoring the second is how a machine with plenty of memory ends up answering too slowly to keep.

The notation on this page, once. A GPU is a graphics processing unit and VRAM is its video random-access memory. B counts billions of parameters, so a 24B model carries 24 billion weights. bf16, FP8, Q4 and AWQ4 say how many bits each weight is stored on: the 16-bit brain floating point training format, 8-bit floating point, a 4-bit quantisation, and activation-aware weight quantisation at 4 bits. GB is gigabytes; GiB is gibibytes, the powers-of-two kind, about 7% larger. tok/s is tokens per second. An MoE is a mixture of experts, a model where each token activates only a fraction of the weights, and a VLM is a vision-language model, which reads images as well as text.

The one fact that decides everything

Generating a token requires reading every active weight out of memory once. So the ceiling on speed is arithmetic:

tokens/second  ≤  memory bandwidth  /  active weight bytes

Published DGX Spark measurements confirm it almost exactly. The GB10 has 273 GB/s: Qwen3 8B at bf16 (~16 GB) is measured at 13.6 tok/s against a roofline of 17.1, and Llama 3.1 70B at Q4 (~40 GB) at 4.4 tok/s against 6.8. This project assumes 70% efficiency, which is conservative against both.

The word that does the work is active

ModelTotalActive/tokenCeiling at 819 GB/s
Devstral Small 2 24B, dense, FP825 GiB25 GiB~23 tok/s
Qwen3-Coder-Next 80B/3B MoE, AWQ445 GiB~9 GiB~62 tok/s
Decode speed against memory bandwidth, for a dense and a mixture-of-experts model Two lines rising from the origin. The mixture-of-experts model, reading about 9 GiB per token, rises steeply and clears 60 tokens per second at 819 GB/s. The dense 24B model, reading 25 GiB, rises about three times more slowly. A dashed red line marks the 20 tokens-per-second comfort floor. The DGX Spark, at 273 GB/s on the dense model, sits below that floor. 0306090120 0273546819 memory bandwidth, GB/s tokens/second (higher is better) 20 tok/s — the comfort floor Mac Studio M3 Ultra same machine, dense model DGX Spark — below the floor Qwen3-Coder-Next 80B/3B, AWQ4 — ~9 GiB active Devstral Small 2 24B, dense FP8 — 25 GiB active
Two conclusions the prose can only assert. The mixture-of-experts line sits far above the dense one, so the best open coding model is also the fastest to serve. And the Spark's bandwidth puts it under the floor whatever it loads — capacity cannot buy its way out of a bandwidth wall.

The best open coding model available is also the fastest to serve. Choosing a smaller model to go faster is, here, exactly backwards — and it is why the recommendation below costs less than it looks.

For a small team, VRAM buys quality

Two people need two concurrent sequences above 20 tok/s. That is a low bar, so the memory should go into a better model rather than into concurrency.

MachineCostSpeedSequencesContext
Apple Silicon you already own (96 GB M2 Max)$027 tok/s5256k
Mac Studio M3 Ultra 96 GB$5,60055 tok/s23256k
RTX PRO 5000 Blackwell 72 GB$12,60090 tok/s16256k
RTX PRO 5000 Blackwell 48 GB$10,200the best model does not fit
DGX Spark 128 GB$6,95018 tok/s0below the floor

The 48 GB NVIDIA card cannot run the best model at all. On NVIDIA the entry ticket is the 72 GB card at $12,600; on Apple it is a $5,600 Mac Studio, because unified memory is cheap and GDDR7 — graphics double data rate memory, seventh generation — is not.

The DGX Spark loses on bandwidth, not on memory. It has 128 GB and serves the best coder at 18 tok/s. Buying more of them does not help: bandwidth does not aggregate across machines for a single stream.

The recommendation

  1. Step zero — free

    Measure the Mac you already have

    The arithmetic says a 96 GB M2 Max already serves Qwen3-Coder-Next at ~27 tok/s with a 256k window. That is a projection, not a measurement, and the difference is this project's entire argument — so pull the model, run a real session, and time a real patch before spending anything.

  2. If it is not enough — $5,600

    Mac Studio M3 Ultra 96 GB

    819 GB/s, twice the M2 Max, ~55 tok/s on the best open coder, holding the 80B model plus an 8B VLM plus the embedder at once. A complete machine: no chassis to spec, no 600 W card, no rack, 270 W and silent.

  3. If you need schema-constrained vision — $12,600

    RTX PRO 5000 Blackwell 72 GB

    The one thing Apple cannot do today. Driving a vision model from an agent means constrained decoding, vLLM has guided decoding, and vLLM has no macOS runtime — so vision + schema ⇒ Linux + discrete NVIDIA.

The cheapest upgrade is not hardware

The sizing is more sensitive to the average live context per session than to almost anything on the invoice — and that figure belongs to the harness, not to the card.

23

sequences on a Mac Studio M3 Ultra at 32k live context — bounded reads, an indexed charter, early compaction

11

at 64k — an agent given a window and left to fill it, for identical output

An agent that pastes the whole standards document into every prompt, reads files whole instead of in slices, and never compacts halves the machine for identical output. That is the measurable argument for everything else here.

A note on prices

They moved violently and they are not done. The RTX PRO 6000 launched at $8,565 in March 2025 and sells for ~$16,000 today; the DGX Spark went from $3,999 to $6,950 for its 128 GB configuration. A 2026 sizing done on 2025 prices is wrong by a factor of two. Surveyed 2026-10-19; re-survey before buying.

Read the full study, with every command that produced a figure →