FP8 is supposed to double the matrix throughput of an AI accelerator. It does — and on AMD's Instinct MI355X that doubling frequently fails to reach the user, because attention is not a matrix problem all the way through.

Zyphra's new ZYFA library is a careful accounting of the gap. ZYFA — Zyphra FlashAttention on AMD — is a set of FP8 (e4m3) and BF16 prefill kernels, open-sourced under Apache 2.0 for CDNA4 (gfx950), ROCm 7.2.3 and PyTorch 2.12, with a drop-in `flash_attn_func` interface.

The arithmetic of the stall

Between the two matrix products of attention — QKᵀ and PV — sits a softmax. FP8 doubles the speed of the matrices. The softmax runs on the vector unit and does not care about precision; it costs the same either way.

For a block of 32 queries against 128 keys, the matrix work takes 1,024 cycles. An exact FP8 softmax takes 896 vector cycles. The matrix units therefore have to be overlapped almost perfectly to stay busy — and the MI355X does not track matrix instruction completion in hardware, so the overlap has to be created by instruction order itself. In Zyphra's thread traces of AMD's own AITER library, the matrix pipe sat idle for 31.7% of cycles.

One multiply-add instead of an exponential

The trick doing most of the work is an arithmetic observation. Read an e4m3 bit pattern as an integer and it grows by 8 each time the underlying value doubles. So the code for 2^x is roughly 8x plus a constant — meaning a probability can be written with one multiply-add and one integer conversion.

Zyphra calls this Lean P, and it cuts softmax vector cycles per block from 896 to 384. Combined with a two-wave ping-pong schedule, in which each SIMD alternates between issuing matrix instructions and computing probabilities, the matrix pipe goes from 60% busy to 86% busy in steady state. A third change — scaling each 64-key block of probabilities by a power of two before rounding, then dividing by the sum of the rounded values using one extra matrix instruction — keeps accuracy from collapsing under the cheaper softmax.

What it actually delivers

On captured real activations at 16K tokens and sustained clocks, the FP8 kernels run 1.16x faster than AITER on Mistral-7B, 1.17x on Qwen3-8B and 1.76x on DeepSeek-V2-Lite. On synthetic Gaussian benchmarks at 32K, the head-dim-128 kernel reaches 2,672 TFLOP/s against AITER's 1,944 — 53% of the FP8 datasheet peak versus 39%. At the MLA shape it is 2,464 against 1,237.

Accuracy improves too, which is the part that will decide adoption. At head dimension 128, ZYFA's FP8 error is 62–78% lower than AITER's. Against an fp64 reference the error is 0.22x AITER's on Mistral-7B and 0.38x on Qwen3-8B; full-model KL divergence is 3.4×10⁻³ against 5.9×10⁻³. Outputs were bitwise identical across 7,200 launches.

The honest caveat, and it is a good one

Real-model gains are smaller than the best-case benchmark figure, and Zyphra explains why rather than hiding it. MI355X clock speed depends on how many operand bits flip; AITER lets more small probabilities become zeros or subnormals, which flip fewer bits and let the clock run higher. On the Mistral-7B layer ZYFA runs at 1,628 MHz against AITER's 2,108 MHz — and still finishes sooner. The team's recommendation is to benchmark on real activations at sustained clocks, which is advice that applies well beyond AMD.

The limits are worth stating plainly: causal prefill forward only. No decode, no paged KV cache, MI355X only, ROCm 7.2.3. AMD's software story has been the recurring complaint against its accelerators. This is how such a story gets fixed — one kernel at a time.