NVIDIA NeMo-RL Utilizes GRPO for Advanced Reinforcement Learning
[ad_1]
Peter Zhang
Jul 10, 2025 06:07
NVIDIA introduces NeMo-RL, an open-source library for reinforcement studying, enabling scalable coaching with GRPO and integration with Hugging Face fashions.
NVIDIA has unveiled NeMo-RL, a cutting-edge open-source library designed to improve reinforcement studying (RL) capabilities, according to NVIDIA’s official weblog. The library helps scalable mannequin coaching, ranging from single-GPU prototypes to huge thousand-GPU deployments, and integrates seamlessly with common frameworks like Hugging Face.
NeMo-RL’s Architecture and Features
NeMo-RL is a half of the broader NVIDIA NeMo Framework, identified for its versatility and high-performance capabilities. The library contains native integration with Hugging Face fashions, optimized coaching, and inference processes. It helps common RL algorithms such as DPO and GRPO and employs Ray-based orchestration for effectivity.
The structure of NeMo-RL is designed with flexibility in thoughts. It helps numerous coaching and rollout backends, guaranteeing that high-level algorithm implementations stay agnostic to backend specifics. This design permits for the seamless scaling of fashions with out the want for algorithm code modifications, making it superb for both small-scale and large-scale deployments.
Implementing DeepScaleR with GRPO
The weblog publish explores the utility of NeMo-RL to reproduce a DeepScaleR-1.5B recipe utilizing the Group Relative Policy Optimization (GRPO) algorithm. This includes coaching high-performing reasoning fashions, such as Qwen-1.5B, to compete with OpenAI’s O1 benchmark on the AIME24 tutorial math problem.
The coaching course of is structured in three steps, each rising the most sequence size used: beginning at 8K, then 16K, and lastly 24K. This gradual improve helps handle the distribution of rollout sequence lengths, optimizing the coaching course of.
Training Process and Evaluation
The coaching setup includes cloning the NeMo-RL repository and putting in mandatory packages. Training is carried out in phases, with the mannequin evaluated repeatedly to guarantee efficiency benchmarks are met. The outcomes demonstrated that NeMo-RL achieved a coaching reward of 0.65 in only 400 steps.
Evaluation on the AIME24 benchmark confirmed that the skilled mannequin surpassed OpenAI O1, highlighting the effectiveness of NeMo-RL when mixed with the GRPO algorithm.
Getting Started with NeMo-RL
NeMo-RL is obtainable for open-source use, offering detailed documentation and instance scripts on its GitHub repository. This useful resource is superb for these trying to experiment with reinforcement studying utilizing scalable and environment friendly strategies.
The library’s integration with Hugging Face and its modular design make it a highly effective instrument for researchers and builders in search of to leverage superior RL strategies in their initiatives.
Image supply: Shutterstock
[ad_2]
