Kimi K3, previewed: inside the first open 3T-class model

Community Article

Published July 19, 2026

Upvote

Hannes von Essen

HannesVonEssen

embedl

This Hugging Face edition adapts our original interactive Kimi K3 preview at hfviewer.

Moonshot AI announced Kimi K3 on July 16: a 2.8-trillion-parameter open model with a 1-million-token context window, native vision, and two architectural bets that go beyond scaling up the usual recipe. The weights land on Hugging Face by July 27. This page is our deep-dive, prepared in advance: the benchmarks and the architecture story now, the traced graph the moment the checkpoint drops.

Kimi K3 specifications: 2.8 trillion parameters, top-16 of 896 experts, one-million-token context, a three-to-one KDA to Gated MLA cycle, MXFP4 weights with MXFP8 activations, and a July 27 weights release

The architecture: four ideas carry it

K3 is not a scaled-up K2. Moonshot credits the jump to roughly 2.5× better scaling efficiency than Kimi K2, and the launch post names the mechanisms plainly. Each one lands on a trend line we have been tracking across the 2,400+ models traced on hfviewer.

Kimi Delta Attention · the workhorse

KDA is Moonshot’s delta-rule take on linear attention: short convolutions, L2-normalized keys, and gated state updates instead of a quadratic attention matrix. Three of every four attention layers in K3 are KDA. That is the industry’s loudest trend right now: linear-attention layers appear in 29% of new models in our catalog this quarter, up from near zero a year ago.

Animated comparison between a growing full-attention cache and Kimi Delta Attention's fixed recurrent state. Open linear attention in the hfviewer glossary

Gated MLA · the precise quarter

Every fourth attention layer is full multi-head latent attention with output gating, keeping global, lossless retrieval in the stack while KDA does the bulk work. MLA itself just had its breakout quarter: 23% of new models now ship it, and with K3, Moonshot pairs it with linear attention rather than choosing between them.

Animated Gated MLA diagram compressing full key and value vectors into shared latent slots before an output gate. Open multi-head latent attention in the hfviewer glossary

Attention Residuals · the new one

AttnRes is the genuinely novel piece: instead of every block adding uniformly onto one residual stream, blocks selectively retrieve representations from earlier depths (the α operators in Moonshot’s diagram). Think of it as attention over depth, not just sequence. No open model we have traced does this; it will be a new shape on the graph.

Animated Attention Residuals diagram in which learned alpha weights select and mix representations from earlier model depths. Open Attention Residuals in the hfviewer glossary

Stable LatentMoE · the sparsity bet

Feed-forward blocks route top-16 of 896 experts, the largest expert pool ever shipped in the open. Balancing at that sparsity is its own research problem: Moonshot’s Quantile Balancing derives allocation from router-score quantiles instead of an auxiliary loss, alongside a SiTU activation and Per-Head Muon optimizer. See MoE routing in the glossary for the family this extends.

Animated Stable LatentMoE router sending successive tokens to different sparse subsets of a large expert pool. Open MoE routing in the hfviewer glossary

Animated top-to-bottom Kimi K3 architecture cycle with KDA, Gated MLA, Stable LatentMoE, and Attention Residuals

Left to right: the 3:1 KDA-to-MLA cycle every token flows through · the expert pool one LatentMoE routes over · the α operators retrieving from earlier depths.

The sparsity race, on one chart

Every dot below comes from a graph traced from real weights on hfviewer; K3 is plotted from the launch post. The direction is unmistakable: expert pools keep growing while the active fraction falls. K3 pushes the pool to 896 but activates 16, more than recent frontier MoEs, landing at 1.8% active.

The original hfviewer sparsity chart showing routed expert count against the share active per token

Active share counts routed experts only (shared always-on experts excluded). Sources: traced configs on hfviewer; Kimi K3 from Moonshot’s launch post.

Benchmarks

Moonshot’s own framing is unusually candid: K3 “still trails the most powerful proprietary models” overall while leading everything else. The numbers back that up. Across the 33 public rows in the launch table, K3 is first or tied-first on 8, ahead of every other open model on essentially all of them, and usually within a few points of the frontier. Standouts: SWE Marathon (42.0, two points clear of the field and triple the GPT 5.5 score), BrowseComp (91.2), DeepSearchQA (95.0), and OmniDocBench (91.1).

All K3 scores at max reasoning effort. Harnesses differ per benchmark (KimiCode, Claude Code, Codex) and several third-party scores are cited from external leaderboards or release blogs; Claude Fable 5 was evaluated by a third party and may include fallback behavior. Rows marked cited mix first-party and cited numbers; internal rows are Moonshot-internal. Elo rows use a different scale. Full footnotes in the launch post.

What it will look like on hfviewer

Until July 27, the closest traced relative is Kimi K2.7-Code: an MLA MoE decoder routing top-8 of 384 experts plus one always-on shared expert. Explore it below, then picture the K3 diff: three quarters of the attention layers turning into KDA blocks with their convolutions and gates, the expert pool growing to top-16 of 896, and a web of AttnRes edges that no current graph has.