**[From GPT-2 to Kimi K3: Seven Years of Large Model Architecture Evolution, Essentially a Battle for Memory]** According to monitoring by 动察 Beating, Baseten engineer Ali published a long article dissecting the evolution from GPT-2 to Kimi K3. Based on parameter count, one K3 is roughly equivalent to about 22,580 GPT-2 models. However, he argues that the real change over the past seven years is not merely the expansion of model size but the continuous transformation of how models store, update, and retrieve information. GPT-2 uses full attention. Each token leaves behind its own key and value, which can be revisited at any time. This ensures the most complete information retention, but as the context length increases, the KV cache and computational costs grow significantly. Linear attention compresses the entire history into a fixed-size state. Costs no longer scale with context length, but the trade-off is that different pieces of information are squeezed into the same space. As the content grows, old memories begin to interfere with each other. Subsequent architectural innovations have focused on adding management rules to this limited memory space. DeltaNet allows the model to first read old content and then overwrite it based on the differences between new and old information. Gated DeltaNet introduces global forgetting, enabling old information to decay as the context changes. KDA further refines forgetting control down to individual channels. Different pieces of information can have different retention durations: long-term facts are preserved longer, while temporary instructions and local variables decay more quickly. Ali believes this is the most critical advancement of Kimi Linear. However, fixed states cannot retain all details. Therefore, K3 does not rely solely on KDA but arranges 23 groups of four-layer structures into a "three layers of KDA plus one layer of MLA," with an additional MLA layer at the end. KDA is responsible for low-cost maintenance of long-term memory, while MLA periodically revisits the full context to retrieve precise information such as numbers, code, and early instructions. K3 also introduces Attention Residuals. In ordinary models, the results of each layer are continuously added together, causing information extracted in earlier layers to be diluted by subsequent computations across 93 layers. K3 divides the network into 12-layer blocks, allowing later layers to call intermediate representations formed in earlier stages as needed, without having to re-derive everything from the original input each time. Thus, the key to K3 is not a sudden breakthrough in black-box technology. Its true uniqueness lies in combining recurrent memory, global retrieval, inter-layer revisiting, and expert routing into a hierarchical information system. The evolution of model architectures over the past seven years has essentially been about teaching models how to decide what is worth retaining, what should be overwritten, and when it is necessary to refer back to the original input. [Original Link]