AstraFlow: Infini-AI-Lab's Open-Source Dataflow RL System for Multi-Agentic LLM Training

AIDeveloper44 Team

What Is AstraFlow?

AstraFlow is an open-source reinforcement learning framework from Infini-AI-Lab designed from the ground up for the unique demands of agentic LLM training. Unlike general-purpose RL frameworks, AstraFlow is built around a dataflow-oriented architecture — treating the training pipeline as a graph of composable data operations rather than a monolithic training loop.

According to the team's paper, AstraFlow is the first LLM RL framework to support fully asynchronous multi-policy collaborative training — a capability that becomes critical as agent systems grow more complex.

Key Capabilities

AstraFlow ships with four headline features:

• 2.7× faster multi-policy collaborative RL training — achieves comparable or better accuracy than verl-based baselines while dramatically reducing wall-clock training time
• Zero-code system flexibility — supports elastic multi-policy training and cross-region rollout across heterogeneous GPU configurations without requiring code changes
• ≤1.1% sparse transfer for remote rollout — by syncing only weight deltas rather than full model checkpoints, AstraFlow compresses remote rollout sync from ~28 GB down to ~1.5 GB per step
• Substitutable rollout and trainer services — modular architecture allows swapping out rollout and training components independently, enabling flexible deployment strategies

Why Multi-Policy RL Matters

As LLM-based agent systems become more sophisticated, the most capable architectures involve multiple cooperating or competing policies — a router model, a generator model, a verifier, a critic. Training these systems jointly with RL has been technically challenging because existing frameworks were designed for single-policy settings.

AstraFlow's dataflow orientation makes multi-policy training a first-class concern. The framework natively handles the data routing, synchronization, and scheduling complexity that multi-policy setups require.

The Sparse Transfer Innovation

One of AstraFlow's most practically impactful features is its sparse weight transfer approach for remote rollout. Inspired by Fireworks AI's sparse RL transfer research, AstraFlow only syncs the weight deltas between training steps — keeping synchronization traffic to ≤1.1% of total model weights. For a large model, this means the difference between a 28 GB sync and a 1.5 GB sync at every rollout step.

Getting Started

AstraFlow is fully open-source and available now. Explore the code, paper, and documentation at infini-ai-lab.github.io/astraflow and github.com/Infini-AI-Lab/astraflow. The accompanying paper is available on arXiv (2605.15565).

Enjoyed this?

Get more posts like this delivered to your inbox.

Subscribe