
## veRL Background

Since September 2023, ByteDance has developed the veRL reinforcement learning framework internally. After a period of polishing and verification, it was officially open-sourced in October 2024. Once open-sourced, veRL has received extensive attention and high recognition in the open-source community, with both the open-source ecosystem and developer active level being hot. The main achievements of its community are as follows: 
1. **High community star count and contributors**
   1. After six months of open source, it has 8.6k stars and nearly 200 contributors. It is growing rapidly in the Github community. 
2. **Many academic and industrial users**
   The main users and contributors include researchers from top universities at home and abroad and global leading technology companies. Some of the names are as follows: 
   1. PKU, THU, UIUC, UCB, UCLA, HKU, Stanford, Northwestern, MIT, …
   2. Amazon, NVIDIA, LMSys, Alibaba, StepFun, Anyscale, OpenHands, …
3. **Rich extension tools/frameworks for veRL**
   1. Nearly 30 industry tools/frameworks are based on veRL for research work, and most of them have received a good number of stars and attention. These include TinyZero, SkyThought, etc.

<div style="width: 90%; margin: 0 auto; display: flex; justify-content: space-between; align-items: flex-start;">
  <img src="/verl-blog/images/blog/verl_v0_4/repo_data.png" style="width: 32%; height: auto; display: block;">
  <img src="/verl-blog/images/blog/verl_v0_4/awesome_work.png" style="width: 67%; height: auto; display: block;">
</div>

The effectiveness of veRL comes from its technical depth (e.g., Hybridflow architecture, GRPO algorithm), engineering practicality (efficient training and production adaptation), and ecological openness (open-source community and enterprise-level support). Its high star count on GitHub, multiple top-conference papers/extended framework references, and extensive industrial implementation cases all verify its dual influence in the developer community and academia. In the future, with the participation and investment of more developers, veRL is expected to become one of the standard tool chains in the field of large model reinforcement learning. 

verl released version 0.4.0 in June 2025, bringing a series of important new features and improvements around the future needs of reinforcement learning for **stronger tool calls, larger models, and better performance**. This article will introduce the Highlight Features and major changes in this version.

## Stronger Tool Call
By upgrading the Rollout module, Verl unlocks **tool invocation** and **multi-turn conversation capabilities**, enabling the model to call tools for deeper thinking. This upgrade enables the use of programming to assist in solving mathematical problems, such as [Retool](https://retool-rl.github.io/), and the use of  [Search-R1](https://github.com/PeterGriffinJin/Search-R1)  to call search engines during reasoning to improve problem - solving capabilities. 

The following figure shows the call relationship of the veRL Rollout module. veRL already supports tool calls and multi-turn  conversation scenarios, allowing users to customize tools 
- **Search tool**: Search enhanced inference Model Training recipe built on this feature
- **Code Sandbox**: Integrated with Sandbox Fusion Secure Code Sandbox Function
  - Using the **veFaaS** code sandbox service provided by Volcengine, you can horizontally scale the concurrency capabilities, ensuring security while further improving performance
- **MCP**: Use the mcp method to make tool calls, which is convenient for docking with various external services.

<img src="/verl-blog/images/blog/verl_v0_4/rollout_modules.png" style="display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 80%"></img>

## Larger Model
Verl now supports **DeepSeek 671B and Qwen3 235B** large MoE models. It is one of the few open-source solutions that can perform reinforcement learning training on such large-scale models. 
- **Megatron Preview Support**: Use the Megatron backend to support reinforcement learning training for large MoE models 
- **Supported models**: DeepSeek-V3, Qwen3-235B, Mixtral, Moonlight, etc.
- **Parallel Technology**: Support Expert Parallel and Context Parallel
- **Distributed Checkpoint**: Support dist-ckpt function, gradient checkpointing

[Document Training DeepSeek 671b](https://verl.readthedocs.io/en/latest/perf/dpsk.html)

## Better Performance
More efficient system design in both Rollout and Trainer, resulting in significant end - to - end performance improvement. 

### Rollout performance improvement 20%
- **Rollout asynchronous execution framework for load balance**: veRL changes the synchronous rollout method in the batch to an asynchronous method. When the tool is called, the GPU can be prevented from being idle, and the load balance mechanism is introduced to dynamically allocate the load. 
  - **Eliminated unnecessary logprob calculations**
  - **Task scheduling mechanism based on server - base implementation**: Load balance ability among multiple dp ranks to reduce the computing power loss caused by idle machines due to long - tail samples. 
- **Support vLLM and Sglang inference engines**
- **Cross entropy kernel fusion**: Significantly reduce peak memory usage
  - Enable by setting `actor_rollout_ref.model.use_fused_kernels=True`

<img src="/verl-blog/images/blog/verl_v0_4/async_rollout.png" style="display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 70%"></img>

> Testing environment: Hardware: 8xA100, Model: Qwen2.5-1.5B-instruct, Dataset: gsm8k, Batch size 1024, Engine: Sglang
> <img src="/verl-blog/images/blog/verl_v0_4/1_5B_perf.png" style="display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 90%"></img>

> Testing environment: Hardware: 8xA100, Model: Qwen2.5-7B-instruct, Dataset: gsm8k, Batch size 1024, Engine: Sglang
> <img src="/verl-blog/images/blog/verl_v0_4/7B_perf.png" style="display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 70%"></img>

### Trainer performance continues to improve 
- **FSDP2 support**: Offers better throughput and memory usage compared to FSDP1 and is compatible with other features such as torch.compile
  ```txt
  actor_rollout_ref.ref.strategy=fsdp2
  actor_rollout_ref.actor.strategy=fsdp2
  critic.strategy=fsdp2
  reward_model.strategy=fsdp2
  ```
- **LoRA support**: enables training of 70B+ models on a single node with 8 A100 cards
  -  Ability to train ultra-large models (with more than 70 billion parameters); 
  -  Can use a larger batch size; 
  -  The trained model is more convenient for migration and deployment; 
  -  Techniques like [SLoRA](https://arxiv.org/abs/2311.03285) or [CCoE](https://arxiv.org/abs/2407.11686) can be used to run more different large model weights with limited resources.

### Other optimizations 
In Performance optimization, we introduced several key technologies: activating the offload function significantly reduced memory usage; the Ulysses sequence parallel technology was designed specifically for VLM to improve computational efficiency; Reward asynchronous optimization accelerated the training process by synchronously processing the reward value when calculating log_prob; the Ray analysis timeline provided a powerful visualization tool for system performance analysis. These optimizations together formed a more efficient computing architecture and significantly shortened the training cycle.

Important breakthroughs have also been made in deployment and hardware support: the dstack simple deployment solution greatly simplifies the configuration process and enables one - click model publishing; at the same time, it enhances the compatibility with non - NVIDIA GPU hardware, especially the in - depth optimization for AMD graphics cards, enabling more hardware platforms to run our framework efficiently. These improvements significantly lower the threshold for users and expand the scope of application scenarios. 

## New algorithms and recipes
- **PPO and GRPO documents**: Detailed explanation of these two core algorithms
- **DAPO**: Decoupled Pruning and Dynamic Sampling Policy Optimization
- **SPIN**: Self-play fine-tuning
- **Self-Play Preference Optimization**: Self-Play Preference Optimization
- **OPO**: On-policy RL with Optimal Reward Base Line
- **DrGRPO, REINFORCE++, Dual-Clip PPO** and other new algorithms
- **Kimi-VL Example**: Visual Language Model Support 
- **Qwen3 Example**: Support for the latest Tongyi Qianwen model 
- **Video Input Support**: MultiModal Machine Learning Capability Expansion
- **Warmup-Stable-Decay Scheduler**: Improved Learning Rate Scheduling
- **RoPE Scaling**: Optimizing Positional Encoding
- **GPQA and livecodebench evaluation**: New evaluation benchmarks
- **ClearML Logs**: New Experiment Tracking Options


## Future Open Source Plan
Volcano will build the veRL open source community for a long time. Come and use it
1. Better performance and stronger scalability:
  1. Scalability: It can support large models such as QWen3 - MoE 235B A22B and DeepSeek - 671B, with the ability to scale accordingly. 
  2. Better performance: Support Megatron-based training, and use parallel methods such as EP, PP, CP, and Zero1 to increase the training scale. Optimize operators such as entropy loss to improve the performance of the training phase.
2. Support complex tool usage scenarios and unlock more reinforcement learning (RL) capabilities:
  1. Have the infrastructure to use tools efficiently: 
    1. By using the asynchronous call interface of the inference engine, multi-round conversations are supported, and the performance is further improved through methods such as pipeline parallelism and partial rollout.
    2. Provide a registration and construction framework for tools to facilitate community developers to provide new tools. 
  2. Supported tools include:
    1. Code Sandbox
    2. Search Engine
    3. External service based on MCP interface.
    4. etc
3. **Collaborate with external teams to support more scenarios.**
  1. NVIDIA Megatron Team：[https://github.com/volcengine/verl/issues/1033](https://github.com/volcengine/verl/issues/1033).
    1. Future development directions include: supporting FP8 training, optimizing training performance, and supporting sglang and trtllm inference engines. 
  2. SGlang Team：[https://github.com/volcengine/verl/issues/1579](https://github.com/volcengine/verl/issues/1579).
    1. Focus on rollout and tool support. 
  3. more

## Summary
veRL version 0.4.0 brings a comprehensive upgrade, from supporting ultra-large-scale MoE models, to tool invocation and multi-round dialogue capabilities, to optimization for resource-constrained environments. These improvements make veRL one of the most powerful and flexible reinforcement learning frameworks available, providing researchers and developers with advanced tools to train and optimize the next generation of AI systems.

## Reference
[Best Practices of veRL on MLP](https://bytedance.larkoffice.com/docx/D8MTdtPZPo0a8MxkquTcBGM7nwe)

BLOG