$1500 to train 1B basic model from scratch! Sapient Open Source Hierarchical Reasoning Architecture HRM Text According to Beating monitoring, Sapient Intelligence has open-source a text generation basic model HRM Text with 1 billion parameters (1B). This is a pure pre trained model based on the Hierarchical Reasoning Model (HRM) architecture. It reduces the computational power consumption of pre training the basic model by 130 to 600 times by introducing latent spatial inference at the bottom of the architecture. Specifically, HRM Text completed pre training using only 40 billion (40B) structured tokens, with a data volume of approximately one thousandth that of conventional models at the same level. Official testing shows that using two 8-card H100 servers, it takes about 46 hours to complete the 1B version from scratch, with a computational cost of approximately $1472; The 0.6B version only requires a single node to run for 50 hours, with a hardware cost of approximately $800. The complete engineering framework, including data extraction, sequence packaging, and PyTorch distributed training, has been synchronized and open sourced. The support for extreme cost reduction lies in the unique dual timescale recurrent design. The model is equipped with two sets of Transformer modules: fast (low-level) and slow (high-level). These two sets of modules alternate and iterate on the same batch of inputs, exchanging information through state addition. This design allows the model to dynamically expand its computational depth by increasing the number of cycles, while keeping the total number of physical parameters fixed. The cliff like drop in the threshold for pre training has given many model theories that were previously shelved due to high computational power the opportunity for low-cost validation. It should be noted that only unaligned pure pre trained weights are released this time, and the model can only perform prefix continuation tasks and cannot be directly used as a question answering assistant. [Original link]