openfront-rl

PPO policy checkpoints for a self-play agent on OpenFront.io, trained with the Rust oftrain stack in djmango/openfront-ai.

Encoders used by these policies live in a sibling repo: djmango/openfront-tile-autoencoder.

Latest (use this)

Field	Value
Run	ppo_v11
Pointer	ppo_v11/latest.safetensors + ppo_v11/latest.state.json + ppo_v11/manifest.json
Update	2486 (milestone file policy_update2485.* is the last numbered snapshot; latest is one step ahead)
Curriculum stage	23
Approx. env steps	16.2M
Reward / schedule	v10-anti-spiral-v1 / curriculum v10
Weights size	~154 MB (safetensors)

Download the current policy:

huggingface-cli download djmango/openfront-rl \
  ppo_v11/latest.safetensors \
  ppo_v11/latest.state.json \
  ppo_v11/manifest.json \
  --local-dir ./ppo_v11

Or from the trainer / play scripts (they already default to this repo):

# restore into a checkpoint dir (validates manifest)
uv run python scripts/hf_checkpoint_sync.py \
  --checkpoint-dir rust/checkpoints/ppo_v11 \
  --run-prefix ppo_v11 \
  --restore-latest

# live play helper
RUN_NAME=ppo_v11 bash scripts/play_live.sh

Always prefer ppo_v11/latest.* over digging through numbered milestones. manifest.json is the compatibility gate (format=oftrain-safetensors).

Layout

ppo_v11/                         # current training run
  latest.safetensors             # ← production pointer
  latest.state.json
  manifest.json                  # architecture + AE refs + update/stage
  policy_updateNNNN.*            # thinned milestone history
  curriculum_advance_*.*         # snapshots at stage promotions
  curriculum_demote_*.*          # snapshots at stage demotions

ppo_v10/ … ppo_v81/              # prior runs: latest.* + manifest only

Older dense policy_update* histories (every ~5 updates) were pruned — adjacent milestones are nearly identical for most uses. Kept history for ppo_v11 is every 100 updates, plus every 25 in the most recent ~200 updates, plus all curriculum advance/demote snapshots.

Architecture (from manifest.json)

• Policy: oftrain-policy schema v3 — spatial grid tower + player/unit streams, legality-masked discrete actions (full OpenFront intent surface).
• Recurrent: LSTM (hidden_size=512, BPTT 24 / rollout 48), action-outcome-v1 context, reset on episode_done.
• Observation: frozen tile autoencoders (ae_v32_nostatic fine 1/8 + coarse 1/16) with exact-state bypass (diplomacy, scalars, transients).
• Training: PPO + GAE, win-gated multi-map curriculum, native engine (Node hedge optional).

See the living design notes in the git repo: DESIGN.md.

Prior runs

Run	Role	What remains here
ppo_v11	Current	latest + thinned milestones + curriculum snapshots
ppo_v10	Previous mainline	latest.* + manifest.json only
ppo_v9, ppo_v86…ppo_v81	Lineage / ablations	latest.* + manifest.json only
Early ppo_v* / bc_*	Legacy	removed in the HF cleanup (recoverable from git LFS/history only if re-uploaded)

Retention

Going forward, training pods should not re-flood the Hub with every-5-update milestones. Use scripts/hf_prune_openfront_rl.py after large syncs, and prefer syncing latest.* / manifest.json / curriculum markers plus sparse milestones.

License / credit

MIT, matching djmango/openfront-ai. OpenFront itself is a separate project — this repo only hosts learned weights and trainer metadata.