vLLM reports day-0 RL results for Mistral Large 4 with SkyRL

What happened

vLLM reports that LoRA RL on Mistral Large 4’s NVFP4 checkpoint with SkyRL raises AIME 2024 avg@8 from ~64% to ~75% in 75 training steps on 8 NVIDIA GB200 nodes. vLLM generates rollouts, while FSDP2 or Megatron handles training using rank-32 LoRA, NVFP4 quantization-aware training, FP8 KV-cache simulation, routing replay (R3), and token-level truncated importance sampling. FSDP2 maintains a mean absolute trainer–vLLM logprob gap of ~0.02–0.026; Megatron achieves similar accuracy with a comparable gap.

Read next

• More changes in today’s briefingMistral releases improved Mistral Large 4 open weights for coding and cybersecurity

The story so far

vLLM shared a LoRA reinforcement learning recipe for Mistral Large 4 using SkyRL, reporting that it raised the model’s AIME 2024 avg@8 score from about 64% to about 75% in 75 training steps on eight NVIDIA GB200 nodes. The run used Mistral Large 4’s NVFP4 checkpoint. In the recipe, vLLM generates rollouts, while either FSDP2 or Megatron handles training. It combines rank-32 LoRA, NVFP4 quantization-aware training, FP8 KV-cache simulation, routing replay (R3), and token-level truncated importance sampling. vLLM reported a mean absolute gap of about 0.02–0.026 between FSDP2 trainer and vLLM log probabilities, with Megatron achieving similar accuracy and a comparable gap.