LongTraceRL-30B

Model Description

LongTraceRL-30B is a 30-billion parameter (3B active) Mixture-of-Experts reasoning model trained with reinforcement learning on long-context multi-hop QA tasks using trajectory-based tiered distractors and entity-level rubric rewards.

Model Details

• Base Model: Qwen3-30B-A3B-Thinking-2507
• Parameters: 30B total / 3B active (MoE)
• Architecture: Qwen3 MoE (48 layers, hidden size 2048, 128 experts, top-8 routing)
• Training Method: GRPO with entity-level rubric reward
• Context Length: 128K prompt + 32K response
• Language: English

Training Details

• Training Data: 2,815 long-context multi-hop QA samples (LongTraceRL Dataset)
• Training Steps: 200
• Learning Rate: 2e-6 (constant)
• Global Batch Size: 128
• GRPO Group Size: 8
• Rubric Reward Weight (η): 0.3
• Framework: Slime (Megatron-LM + SGLang)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("THU-KEG/LongTraceRL-30B")
tokenizer = AutoTokenizer.from_pretrained("THU-KEG/LongTraceRL-30B")

Citation