ABOUT

verl is a flexible, efficient and production-ready RL training library for large language models (LLMs).

verl is the open-source version of HybridFlow: A Flexible and Efficient RLHF Framework paper.

verl is flexible and easy to use with:

• Easy extension of diverse RL algorithms: The hybrid-controller programming model enables flexible representation and efficient execution of complex post-training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code.
• Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc
• Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
• Ready integration with popular HuggingFace models

verl is fast with:

• State-of-the-art throughput: SOTA LLM training and inference engine integrations and SOTA RL throughput.
• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

Key Features

• FSDP, FSDP2 and Megatron-LM for training.
• vLLM, SGLang and HF Transformers for rollout generation.
• Compatible with Hugging Face Transformers and Modelscope Hub: Qwen-3, Qwen-2.5, Llama3.1, Gemma2, DeepSeek-LLM, etc
• Supervised fine-tuning.
• Reinforcement learning with PPO, GRPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, DrGRPO, etc.
  • Support model-based reward and function-based reward (verifiable reward) for math, coding, etc
  • Support vision-language models (VLMs) and multi-modal RL with Qwen2.5-vl, Kimi-VL
  • Multi-turn with tool calling
• LLM alignment recipes such as Self-play preference optimization (SPPO)
• Flash attention 2, sequence packing, sequence parallelism support via DeepSpeed Ulysses, LoRA, Liger-kernel.
• Scales up to 671B models and hundreds of GPUs with expert parallelism
• Multi-gpu LoRA RL support to save memory.
• Experiment tracking with wandb, swanlab, mlflow and tensorboard.

Citation and acknowledgement

If you find the project helpful, please cite:

• HybridFlow: A Flexible and Efficient RLHF Framework
• A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization

@article{sheng2024hybridflow,
  title   = {HybridFlow: A Flexible and Efficient RLHF Framework},
  author  = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2409.19256}
}

verl is inspired by the design of Nemo-Aligner, Deepspeed-chat and OpenRLHF. The project is adopted and contributed by Bytedance, Anyscale, LMSys.org, Alibaba Qwen team, Shanghai AI Lab, Tsinghua University, UC Berkeley, UCLA, UIUC, University of Hong Kong, ke.com, All Hands AI, ModelBest, OpenPipe, JD AI Lab, Microsoft Research, StepFun, Amazon, Linkedin, Meituan, Camel-AI, OpenManus, Xiaomi, Prime Intellect, NVIDIA research, Baichuan, RedNote, SwissAI, Moonshot AI (Kimi), Baidu, Snowflake, and many more.