A 27B Model in 5.9GB: How Ternary Quantization Stays Standing
An old 11GB card — a GTX 1080 Ti from 2017 — can now run a 27B model at 20 tokens per second.
This is not a joke. The model has been discussed over and over in open-source circles lately: Ternary Bonsai 2 27B. All files total 5.93GB, less than 1/9 of the full-precision model of the same family, and the official claim is 98.2% capability retention.
How does it pull that off? In one sentence: it compresses the model's weights down to three values — -1, 0, +1.
This article covers two things: why this "ternary quantization" approach works (rather than collapsing the moment you squeeze it, as earlier extreme quantization did), and what state it is genuinely in right now — including my hands-on test of running it on a DGX Spark.
1. First, what "ternary" actually means
A large model is filled with tens of billions of "knobs" (weights). Normally each knob is recorded as a 16-bit float — a 27B model needs 53.8GB of memory for the weights alone.
Ternary quantization does one thing: it cuts the scale on every knob down to three settings — -1, 0, +1.
That sounds absurd, but a quick calculation explains the motive: one ternary value carries about 1.58 bits of information; counting the overhead of scaling factors, roughly 1.76 bits on average. Compress 27B parameters that way and you land in the 5.9GB range (full precision needs 53.8GB). That is the arithmetic behind the 1/9 size.
And it can still see images: the model ships with a 0.47B vision tower, so multimodal ability was compressed in along with everything else.
"Extreme compression" is not new. BitNet proposed ternary training back in 2023. But earlier routes either required training a large model from scratch, or produced a model that got dumb after compression — unable to do math problems, unable to write a coherent long sentence.
This time is different: PrismML (a Caltech team) compresses an already-trained Qwen3.8-27B directly. It is not building a new brain; it is putting an existing brain on an extreme diet.
Which raises the question: compress directly, so why doesn't it collapse?
2. Why it works: three key design choices
① Flatten first, then compress
The biggest enemy of extreme quantization is the outlier: a tiny number of weights with extremely large values. Under ternarization those spikes either hijack the whole range or get chopped flat — either way the model distorts.
The official fix is clever: before compressing, apply a Hadamard rotation to the weights.
Picture rugged terrain. Lay three tiers of steps on it directly and nothing fits. But look at the same ground from a different angle (apply an orthogonal transform) and the bumps spread evenly across every direction — the terrain flattens out, and now the steps fit everywhere.
That step is also lossless: the rotation only changes the coordinate system you keep the books in, so no information is lost; at inference time a matching fast transform turns it back, costing only O(n log n) — far less than the main computation, essentially free.
How direct is the effect? A controlled test showed a community 2-bit-class quantization (IQ2_XXS) collapsing to 57.5 on math problems; this one reached 95.8.
② Every 128 weights get their own ruler
Ternary keeps only the direction (positive / negative / zero); the actual magnitude is restored by a scaling factor. The approach here: group 128 weights together, and each group shares one FP16 scaling factor to restore magnitude.
It is like grading a class on only high / medium / low, but drawing the lines group by group — so a few big values cannot hijack the whole range. Final result: 1.76 bits per weight on average.
③ Custom kernels that compute directly on ternary values
If every inference pass had to unpack ternary back into floats first, the memory and time you saved would be handed right back.
So the team wrote custom kernels for Nvidia GPUs and Apple silicon, computing directly on the packed format. There is one trap here that I hit in testing: you must use the official llama.cpp fork — the upstream version either refuses to load, or, more dangerously, loads silently and then emits garbage.
"Flatten first, then compress": rotation evens out the spikes so ternarization doesn't collapse
Three things together: rotation makes it quantizable, grouped scaling preserves accuracy, custom kernels preserve speed. Only then did ternary quantization move from a paper to something usable.
3. Current state: what it can and cannot do
Bottom line first: it works, and the barrier to entry is surprisingly low.
The model is open source (Apache 2.0), downloadable from both ModelScope and HuggingFace; two GGUF tiers (5.95GB / 7.21GB), deployable via llama.cpp, MLX or CUDA.
Ternary 27B at a glance
On quality, the official average across 20 benchmarks is 83.9, retaining 98.2% against the full-precision baseline — and on some items (instruction following) it even comes out ahead. The team also proposed a new metric, "intelligence density" (score converted to an error rate, then divided by size): it is 8.7× that of the FP16 model.
My own benchmarks (DGX Spark):
Benchmarks on this machine (2026-09-26)
- Single-stream decode 29.31 t/s, almost exactly matching the official 29.2;
- 8-way concurrent aggregate 100.8~102.9 t/s — enough for a small team to share;
- About 10.4GB of memory and 62W of power — viable for 24/7 duty;
- Spot-checked math counting, multithreaded code, instruction following and image recognition — all passed.
Speed landscape: RTX 5090 peaks around 143 t/s; an old 11GB 1080 Ti does 20.5 t/s; M5 Max about 47 t/s. The bar for "running a 27B locally" has shifted from a memory arms race to compression efficiency.
But don't look only at 98.2% — the weak spots deserve clarity:
First, long-horizon tasks take a hit. The team measured long-chain tasks for the first time: Terminal-Bench 2.1 scored 52.8 (full precision 69.7) and SWE-bench Verified 60.8 (full precision 80.6) — roughly three-quarters of full precision. The reason is not complicated: small per-step errors accumulate over a long chain, like a game of telephone — one round is fine, twenty rounds and the message has changed. Chat, knowledge Q&A and single-step tasks are fine; as the "brain" of a long-chain agent, it is still one step short.
Second, one small misconception to correct. Some articles say it ships with DSpark speculative sampling for a 2.45× speedup — I checked the official repo: that was a capability of the previous generation; Bonsai 2 has no official drafter yet.
Third, an open question. The team does not say whether the compression is PTQ (post-training quantization) or QAT (quantization-aware training). That determines whether the method can be "picked up and applied" to other models — if it is pure post-training processing and still holds 98.2%, the significance is much larger.
4. My read
The most noteworthy thing here is not "another open-source model," but the signpost it plants for a new track: once a 27B fits on an old 11GB card, the definition of the bar for "running a large model locally" changes.
Who it suits: people with limited memory who want a large model on their own machine; scenarios needing always-on, offline, low-power operation. Who it does not suit yet: people expecting it to fully replace full-precision models on long-chain tasks.
I will be watching three things: whether the Qwen family follows with ternarization; whether this is PTQ or QAT; and whether long-horizon performance can be closed. Those answers decide whether it is "a handsome technical sample" or "the new default for local deployment."
Q&A box: if I run experiments with it on this machine, which would you most want to see?
- A: Wire it into a game as the "brain" of NPCs
- B: Use it as a coding assistant for a week and see if it survives real work
- C: A head-to-head comparison with the 4-bit quantized version on the same questions
Drop a letter in the comments; whichever gets the most votes goes first.
——
I'm Cosmocat: one person, one DGX Spark, that's the whole company. "Putting large models into small machines" is what I've been working on lately: this 27B is already running here — 29.31 t/s, 10.4GB of memory, 62W. If you want to see what I do with it, say so in the "Q&A box."
Note: official benchmarks and speeds are the official figures (as of 2026-10-16); the numbers here are my own records on a DGX Spark (2026-09-26). Ternary quantization is still early — treat the numbers as a starting point and test for yourself.
Sources and notes
- All figures and conclusions in this article come from my own testing or public disclosures; where they differ from the original, this page takes precedence.