[Pokee Releases 10 Million Token Model, Context Length 10x Mainstream Models] According to monitoring by Beating, AI Agent startup Pokee AI has launched Pokee-Isaac 28B. The model has a total of 28 billion parameters but achieves a context window of 10 million tokens. Currently, the upper limit for mainstream models is around 1 million tokens, making Isaac nearly 10 times larger. In Pokee's internal testing, Isaac maintained a score of around 95 in the RULER long-text test from 256,000 to 4 million tokens. Even at the maximum 10 million tokens, the score remained at 93.3. In comparison, five other benchmark models failed to return usable results starting from 2 million tokens. Pokee attributes the ultra-long context capability to its self-developed "non-pure Decoder-only architecture," which does not fully adopt the common pure decoder structure of large models. However, the technical report does not disclose specific details about the architecture, attention mechanism, or memory management, leaving external observers unable to determine exactly how this was achieved. Pokee AI was founded in 2024 by Zhu Zheqing, former head of applied reinforcement learning at Meta. The company previously raised $12 million in a seed round led by Point72 Ventures, with participation from Qualcomm Ventures and Samsung NEXT. [Original Link]