News from the IT community on October 15th: Step-by-Step officially releases the weights of Step5Preview today. This release includes the model weights of BF16 and FP8, as well as a technical blog and accompanying open-source development tools.

According to official introductions, Step and Preview are designed for long-term Agent tasks, adopting a sparse MoE architecture. The total number of parameters is 600B, with each token activation parameter being only 27B. It supports up to 1 million token context windows and natively supports both text and visual input. It performs exceptionally well in scenarios such as AI programming, software engineering, professional knowledge work, and finance. Recently, Step and Preview have ranked second globally in both the weekly and daily usage volume charts.

Currently, the model weights Hugging Face and ModelScope have been released, and support for adaptation to various chip platforms and inference frameworks is provided. Developers can deploy API services that are compatible with the OpenAI interface through vLLM or SGLang. They can also use Transformers for offline inference, integrating the model into their own data, tools, and systems to build agents, applications, workflows, and services.

IT Home has noticed that the officially synchronized and publicized technical blog introduced how Step and Preview can support longer and more complex Agent tasks through the collaborative optimization of architecture and training. Key designs include:

• 92-layer narrow and deep architecture: Provides a deeper computational path for implicit multi-hop reasoning, while controlling the parameter scale and computational cost.
• Sparse GQA: By merging block-level indexing with sparse computing, the computational overhead for processing millions of token is reduced. Among them, using 8 token block-level indexes can reduce the indexing and selection overhead to 1/8 of the original level.
• More stable training: Maintaining the FP32 precision in key states and parameters, combined with fine-grained gradient clipping to mitigate abnormal activation growth and gradient spikes in deep networks.
• Million-level verifiable environment: Covers software engineering, scientific reasoning, machine learning research and development, and professional knowledge work, while continuously calibrating the difficulty, diversity of tasks, and evaluating reliability.
• Reinforcement Learning for Long-Term Tasks: Incorporating Context Compression into End-to-End Policy Training to Enable Agent to Learn to Organize Task States and Continue Execution; Combining with More Precise Reward Attribution to Improve the Learning Effect of Long Trajectories.