One of the details that has caught X community's imagination is that Kimi K3 has constant state. In other words it does not have linearly growing KV cache as the context grows, or your conversation lengthens (for the most part - we will see details and caveats soon).
Long context use cases - e.g. agentic coding - has increased memory demand massively, resulting in 10 to 30 times increase in prices of memory stocks. If you don't require as much memory to store KV cache, is this price increase justified? There is a vibrant debate going on this topic on X.
The detractors say Hybrid Linear attention used by Kimi K3 called Kimi Delta Attention (KDA) is around for quite some time. So it is already factored in prices of memory stocks.
However, they miss the point that - unlike previous attempts - Kimi K3 proved that Hybrid Linear could work so well at 1T+ parameter scale and deliver frontier performance. **This is NET NEW information and deserves serious attention. Before Kimi K3 Hybrid Linear was believed not to be a technique for frontier but just a neat trick used by likes of
**- Qwen (who never quite get to the frontier) or
- Nvidia (who tries everything under the sun, just because they have infinite GPUs).
Let us dive in!
(I want to make it clear - THIS IS NOT INVESTMENT ADVICE).
Brief history of Hybrid Linear:
Linear models always held great promise on paper just like Communism.
Instead of constantly growing KV cache, as context grew; they have a constant state that gets updated. The state size is constant whether you have thousand tokens or a million tokens.
It sounds to good to be true, because it is! Linear models have issues with recalling content from earlier parts of conversation perfectly - e.g. telephone number from earlier part of the conversation.
Why? This is because such models are constantly updating their internal 'summary' (called state) of the context. However, since this context/summary is fixed in size (e.g. 128\128 in case of Kimi K3, per head), its storage capacity is limited. It compresses information to be efficient (that is what the model learns during its training), but it has to drop some information for sure. In other words, the compression is lossy.*
One way to solve the problem is to go for 'hybrid linear' where you combine linear layers (typically 75%) with full attention layers (25%). Hybrid Linear can have efficient version of full attention as well (e.g. DeepSeek style MLA in case of Kimi K3). While this was attempted, the earlier hybrid linear models did not have performance at the frontier level!
What Kimi K3 showed is that - with their variation called Kimi Delta Linear Attention (KDA) - you can achieve near perfect recall, as good as, that of models using full attention in all layer. This is a massive breakthrough, whatever you have been given to believe about it....
How did Kimi K3 achieve it?
Moonshot are not first to use hybrid linear, but they came up with a variation of it that is highly effective. It allows fine grained control over memory decay.
Off course they did many other things, involving data engineering, kernel development etc., use of attention residuals, stable latent MoE etc. (we will understand them towards the end). But they did something seemingly simple that resulted in exceptional performance gain on the attention part.
The model's KDA state can essentially be thought of as a stack of memories (over simplified version). At every new token position a diagonal matrix is computed and the state is multiplied with it. This results in forgetting some memories more and some memories less (jargon: per channel decay). In earlier versions you would forget all the memories to equal extent. This trick of selectively and judiciously forgetting significantly improved the performance!
This is a beautiful visualisation that shows how state matrix updates happen. I will have another blog coming that explains, how this leads to the holy grail "continuous learning".
How much memory saving is there with Kimi K3?**
K3 has 93 layers and 96 heads. KDA has 128 dv and dk dimensions.
Let us see what it takes to process 128K tokens and 1 million tokens in terms of state and KV storage. Let us see two cases:
1) 93 MLA full attention layers (All-MLA) vs
2) 69 KDA + 24 MLA full attention (actual Kimi K3 model as depicted above)
At the bottom of the article I have posted the calculations.
**Key observations:
**1. For All-MLA, KV cache storage grows from 13.7 GB to 107 GB from 128K to 1M tokens.
- As you can see, KDA state for Kimi K3 is constant at 0.22 GB regardless of the context length. Only the MLA part grows from 3.5 GB to 29 GB from 128K to 1M tokens context.
In both cases, there is saving of about 73%. The beauty is KDA does not compromise on frontier performance while offering these saving. MLA is already a massive saving over MHA and GQA (almost 90%), KDA save 73%!
Some will argue savings with DeepSeek V4 architecture at Kimi K3 scale would be even higher. True, DeepSeek V4 is not at the frontier yet. And even if it reaches the frontier, it will improve in exactly same direction: less storage.
This beautiful visualisation produced by Kimi K3 itself shows the savings:
Is Kimi K3 unique in the industry when it comes to such dramatic memory saving?
DeepSeek V4 (https://arxiv.org/pdf/2606.19348v1) has done exceptional work on cutting KV cache by 98%, but nobody considers them near frontier. When DeepSeek V4 was launched did you observe the same level of activity on X?
Also, for all their brilliance all the layers in DeepSeek V4 are either CSA (Compressed Sparse Attention) of HCA (Highly Compressed Attention) - both types are compressed along the sequence dimension and it is lossy. There is not a single layer that has uncompressed information, that is all the layers are lossy - recall in Kimi K3 25% of layers have full attention.
My hypothesis is that Kimi K3 will have better ability to retrieve specific information from earlier context in long context scenarios. Hence, I foresee Kimi K3 architecture will be widely adopted by other labs as well (e.g. ZAI, MiniMax, Qwen, Tencent, Xiomi etc.), just like DeepSeek V3 (MLA) and later DeepSeek V3.2 (MLA + DSA) became widespread in the industry.
DeepSeek's CSA. Notice the compression along sequence dimension. It results in 1 compressed tokens for every 4 tokens.
**
DeepSeek's HCA. Notice the compression along sequence dimension. It results in 1 compressed tokens for every 128 tokens.
What about Funda AI's point that you need to cache KDA state multiple times, as it is not additive like KV cache? Also, will subagents not need KV cache storage?
Recall the previous example, total KDA state across 69 layers is 0.22 GB. So, even when you cache/snap-shot it at every 20K tokens during a long session, it is 50 times across 1M context.
So theses 50 spanshots will take 50*0.22 GB. It all adds up to just 11GB - total l (29.22 GB + 11 GB) stays well below 107 GB required for All MLA at 1M tokens context.
Second, Funda AI's point was that main agents can spawn subagents. People will spawn many subagents. But typically, once a subagent returns the control to main again, its job is done. Its KV + state is deleted. So even if people launch agents that launch hundreds of subagents, KV + state needs are not long term. The main session / main agent's state needs to be preserved much longer; and this where KDA saves a lot!!!
There are patterns where there are long running async agents, in those cases their KV + state would need to be maintained. But I see those patterns implemented rarely. This is not to say it won't change in future.
Synchronous subagents (a very common pattern):
Asynchronous subagents (a relatively rare pattern)
Will KDA's adoption then will have downward impact on memory demand?
I am not a fortune teller, but there is a small and non-zero probability of it busting memory bubble.
First why it may not have any impact on memory demand:
- While savings are real, if open source AI gets widely adopted, most deployments are not going to be as efficient as they are at closed frontier labs. Hence, any gains at the model level will be lost at industry level. We will have large number of small deployments that are not very efficient. E.g. for most enterprises & governments use happens across 12 hrs in a day. What their open source model infra and its memory going to do at night? Infra at large labs is used 24*7 due to global customer base.
- Also, we are making a big assumption that leading labs already don't have similar approaches. OpenAI, Anthropic, DeepMind employ some of the smartest people in the world. So there is quite good chances that such innovations are already in use and have been factored into the memory demand.
That said every once in a while there is a SpaceX that comes and shows the industry leaders how their approaches were inefficient. They think from the first principals. Incumbents - due to organization structure issues, or due to not being hungry enough, do not innovate on key dimensions. History of capitalism is filled with thousands of such examples....
So it is an unlikely scenario - but still with some non-zero probability - that if large labs were not as starved for resources, they may not have explored the entire design space to reduce memory demand to such an extent.
In the AI industry itself, it has happened many times before where big labs have shown gross incompetence:
- XAI's training pipeline had poor MFU (GPU utilisation) - due to habit of throwing compute at the problem, plus other cluster misconfiguration and organization level issues.
- Despite all the resources at their disposal, Meta did train a sh*tty version of DeepSeek's MoE for Llama 4.
- Google took 1.5 years since ChatGPT launch to bring Gemini to the forefront and it is still no one's favourite model for the most popular use case: AI coding.
Constraints force innovation, so it is not unlikely that small highly well organised teams outshine large resource rich teams. And now that K3 is out and its architecture is well know, if KDA like technique - that is now successful at the frontier scale - is adopted by closed labs too; it means 75% reduction in memory demand at least for a few labs, and they are quite big individually, so the impact could be in tens of billions.
These memory savings could be eaten up with increased use due to reduction in cost for sure (Jevon's paradox), but it is not guaranteed that consumption growth will always outstrips the savings....
Diffusion of other innovations: Kimi K3's major innovation is not KDA alone!
Kimi K3 gets rid of positional encoding completely: Positional encodings inform what is sequence number of each token in the context. Its KDA layers, due to their recurrent nature do not need position encodings. They get rid of them even for MLA layers and it does not hurt. Benefits? It does not any special effort to extend model context length compared to earlier models. Elegant.
Kimi K3 implements attention residuals at 2.8T scale: It ensures, effectively, that higher layers are able to pay selective attention to inputs from lower layers. In earlier approaches higher layers could not distinguish between inputs from lower layers at a given token position, as all the values used to be aggregated. This new approach gives the model much higher discernibility. (https://arxiv.org/pdf/2603.15031
Kimi K3 implements Nvidia's Stable Latent MoE: The main benefit of a stable latent Mixture-of-Experts (LatentMoE) architecture is higher model accuracy and more expert capacity at the same or lower inference cost. It achieves this by projecting token representations into a smaller latent space before running expert routing and computation. They picked if from Nvidia's Jan 2026 paper (https://arxiv.org/pdf/2601.18089v1
With all these innovations put together, MoonShot team achieve 2.5 training efficiency over their previous architecture Kimi K2. This is to say, they achieve same validation loss at 2.5 times less FLOPs. That is massive saving even for training compute!!!! This will allow many more labs to train such high scale models with limited compute.....
With K3 Moonshot unlocked new scaling laws. Kimi K3 achieves 2.5× gain in scaling efficiency over Kimi K2
MoonShot has also open sourced their FlashKDA GPU kernel for fast computation of KDA state and MoonMoE for dispatch and combine of tokens. They are pretty confident in sharing so much knowledge, because, they are already onto the next set of innovations that will take then even futher (at the end of day, they too are capitalists).
Scaling laws are not laws of nature, we can always discover better
Furthermore, we may discover newer scaling laws with newer architectures and training paradigms, offering even denser intelligence. Scaling laws are statistical observations, they are not laws of nature that can not be changed. What this means is that with newer models, for smaller size (and lower KV and state) we can have better performance. So demand explosion does not mean running the same large models forever. Models will continue to get denser in terms of intelligence.
We have an ecosystem in form of Chinese labs that is highly constrained, and highly capable - a very potent mix, as discussed. They are discovering corners of the design space that have not been discovered by more well funded labs. They make rapid progress due to open sharing of knowledge. This saves others from wasting their efforts and reinventing the wheel. The evolution of architecture happen much more rapidly than otherwise....
Also, this success by MoonShot team is sure to inspire other equally capable teams across DeepSeek, ZAI, MiniMax, Tencent, Alibaba, ByteDance SEED and even emerging players like Baidu, Ant Ling, Xiomi Mimo, Meituan and many others to be the best, not just as an open source model, but to be the best overall.
Lot of new innovations will happen across emerging labs in the west, as well, whether it is at xAI or MSL now they have gotten their act together and started delivering great models.
That said, as I keep pointing out, China has most number of young AI researchers based on NEURIPS statistics. Contributions in this field come disproportionately more from younger researchers (the source thread), so many more breakthrough are likely to emerge from that part of the world.
China also has 3 major urban centers with AI talent density rivaling that in the Silicon Valley, they act as idea mixing crucibles (the source thread).
Banning open source AI will not help:
An unfortunate conclusion of this discussion may be that we must ban open source AI to protect the western markets. But that would be highly counter productive.
Whether open source AI is banned or not in the western world, these design spaces will get explored, and papers will get published. West also has lot of new great labs emerging. They will discover lot of new ideas too, now that it has been realised there could be lot of efficiency gains that are still possible.
The best thing is to encourage and adopt these innovation and be responsible with investments and avoid wild speculation.
The future is very very bright due to this technology, but we have to be careful not be crazy speculators who get margin called.
We haven't been using as much AI as we could be: whether it is in corporate world, education, research or entertainment.
- While coding has been the the mainstay of GenAI token use, applications like CoWorker have just begun.
- I would love for every student in India to have unlimited access to Opus 4.8 or Kimi K3 level models. They do have access, but to weaker models only.
- Furthermore, we could have much higher value use cases of AI in Science, Engineering and Medicine.
- Also, we have gotten an aging world that we need to care for. Ai is going to be greatly beneficial there. There exist so many possibilities across entertainment and gaming as well. As Jensen Huang mentioned: "every pixel we see on the screen for entertainment will be generated".
Conclusion:
Stock market crash of 1929- the Great Depression - was not a result of demand destruction. It was a result of speculative excesses. The whole world sufferred for two decades. It arguably led to rise of Fascism and WW2.
**
While the future is uncertain, what is clear in my observations is that, AI bulls and memory bulls, in particular, do not factor in algorithmic innovations and discovering of newer scaling laws. For them everything gets eaten by Jevon's paradox, hence not interesting. This attitude has percolated in retail investors as well and has led to insane behaviour that has potential to bring down economies of several countries. **I want to introduce new angles into their analysis. They have far better information on supply chain situations and corporate earnings. I want to arm them with more technical perspective.
To crash memory & AI stocks, we don't need massive reduction in demand, as Jaya Gupta has pointed out. Even small reduction in demand can start a game theoretical competition wherein existing bag holders start booking profits with intention to buy stocks later, this can crash the whole market. And this seems to have begun.....
It is important to not let such large bubbles form and burst catastrophically for the society. Overall I stay very optimistic about potential of this technology (and even memory demand) and future infra build out. It is the most exciting time to be alive!
(NOT INVESTMENT ADVICE)
Appendix
Detailed calculation of memory savings:
To show how good Kimi K3 is, I got K3 itself to do these calculation I have verified they are correct. As mentioned earlier we consider two cases:
1) Kimi K3: 69 KDA + 24 Gated MLA full attention vs
2) Full MLA: 93 MLA full attention
