Efrain Garay · Measured

NVFP4 vs FP8 vs GGUF on a 16 GB Ada GPU: NVFP4 decodes 2.3x faster and ties BF16 on GSM8K

Efrain Garay — Sun, 18 Oct 2026 01:59:59 GMT

When you type «nvfp4 vs» into Google it completes with «fp8», «gguf», «q4_k_m» and «bf16». The usual answer compares different models quantized by different people. On October 16 NVIDIA released Nemotron Labs Dory 1.0 4B in BF16, NVFP4, MXFP8 and GGUF at once, so I measured the same model in every format on an RTX 4070 Ti SUPER (Ada, 16 GB, no FP4 hardware), with the rules written down before the first run.

Format	Engine	Generate (tok/s, batch 1)	Read 2048-token prompt (tok/s)	Weights
BF16	vLLM	51.0	7,003	7.69 GiB
FP8 (per tensor, W8A8)	vLLM	78.5	10,118	4.67 GiB
MXFP8	vLLM	81.4	6,061	4.76 GiB
NVFP4	vLLM	116.8	6,978	3.28 GiB
GGUF Q5_K_M	llama.cpp	58.8	1,710	3.42 GiB (file)

Among the vLLM formats, generation follows weight size. NVFP4 was 2.3 times faster than BF16. On Ada it runs through a weight-only Marlin kernel: the weights stay small, the math stays in 16 bits.
Reading a long prompt, FP8 wins, because it is the only one of these formats whose multiplications Ada does in 8 bits.
Quality, full GSM8K (1319 questions, 8192-token cap): BF16 1249, FP8 1241, MXFP8 1250, NVFP4 1238. All three tie with BF16 under the ±3-point rule; their 95 % intervals stay within ±2 points.
Token by token it is another story. On fixed English and Spanish texts NVFP4 picks BF16's next token 89-90 % of the time, against 94-96 % for FP8 and MXFP8, while perplexity barely moves. MXFP8 is the most faithful to BF16.

The post also covers what broke before measuring (the llama.cpp converter rejects the public checkpoint, a prefix cache that inflated prefill, truncated reasoning being scored as answers) and a mistake the audit caught in my own first fidelity run.

Read the full measurements, the pre-registered sheet and the scripts → https://efraingaray.com/en/blog/nvfp4-vs-fp8-gguf-ada/

Kandinsky 6.0 on a 16 GB GPU: the official recipe runs out of memory, and only the 29B model passed my Spanish test

Efrain Garay — Wed, 14 Oct 2026 18:30:59 GMT

Kandinsky 6.0 generates five seconds of video with its audio, voices included, from a text prompt (MIT license). I wanted to know two things: does it run on a 16 GB card, and can it speak Spanish?

Memory. On a rented 16 GB RTX 4080, the README recipe (per-component CPU offload) ran out of memory with the distilled Lite model; the Qwen2.5-VL text encoder files alone add up to 15.45 GiB. An 8-bit text encoder and sequential offload failed too. What worked was a two-phase recipe: compute the prompt embeddings first, streaming the encoder through the GPU in groups of layers, then drop it and generate with a tiled video VAE. PyTorch allocated at most 11.24 GiB; a vertical clip took about 141 s; the process reached 45.9 GiB of system RAM.

Speech. Eight scenes with one Spanish line each, two seeds, scored with faster-whisper large-v3 against the requested line. The rule, written before running: median word error ≤ 0.20 and Spanish detected in ≥ 12 of 16 clips.

Model	Median Spanish WER	Clips with WER ≤ 0.20
Lite distilled (3B, 10 steps)	1.06	1 of 16
Lite undistilled (50 steps, guidance 5)	0.75	1 of 16
Pro distilled (29B, 10 steps)	0.15	9 of 16

Only Pro met the rule, and its process reached 120 GiB of RAM at roughly 12 to 23 minutes per clip (RTX 3090 with PyTorch memory capped at 15.57 GiB, transformer streamed block by block). The data is compatible with an advantage of the bigger model, but it cannot separate size from distillation.

The full write-up has the recipes, the component diagram, clips with what Whisper heard, the threats to validity (three external audits), and a demo: the small model's video with a replaced voice and the mouth redrawn by LatentSync 1.6.

Read the full measurements → https://efraingaray.com/en/blog/kandinsky6-video-audio-16gb/

Scripts, the pre-registration sheet and the raw JSONL: gist.

Qwen-Image 2.1 on 16 GB of VRAM, quantized to NF4: a real benchmark against FLUX

Efrain Garay — Mon, 12 Oct 2026 03:56:56 GMT

On a 16 GB RTX 4070 Ti SUPER, Qwen-Image-2.1-Turbo does not fit in bf16 (the pipeline is 32.5 GB). Quantized to NF4 with bitsandbytes it does, and I ran it against the FLUX.1-dev Q6_K + pixel-art LoRA that made my blog covers: same prompts, same seeds, same card, every image checked by eye.

What I measured

Speed: 8.4 s per image for Qwen NF4 (8 steps) against 61 s for FLUX + LoRA (30 steps), median, 1344×768. About 7×.
Text in the image: 13 of 18 requested texts exact for Qwen NF4, 7 for FLUX. Unquantized Qwen on a 48 GB A6000 got 17.
Cover prompts: Qwen NF4 drew the requested scene in 11 of 15 covers; FLUX + LoRA in none (6 almost, 9 missing the main action or object).
Memory: 11.9 GiB over the card's baseline for Qwen NF4 (nvidia-smi), 13.1 GiB for FLUX.

What broke on the way

An 8-bit (LLM.int8) transformer returns a near-uniform purple field. NF4 works.
Decoding the VAE in tiles (needed to fit) left faint vertical purple streaks. Fix: make the latents first, drop the transformer and the text encoder from the GPU, then decode the whole frame (0.77 s, 7.1 GiB peak).
An automatic reader (gemma3:4b) accepted misspelled text, and an external audit caught five of my own verdicts. Every number in the post comes from the corrected image-by-image review.

What I lose: the LoRA's atmosphere. Qwen follows the scene but on a flatter background. I switched anyway: every cover on the blog is now generated by Qwen NF4.

Read the full measurements, the side-by-side galleries and the code → https://efraingaray.com/en/blog/qwen-image-vs-flux/

Scripts, prompts and the per-image review: gist.