gpt-2-wikitext-chunks-epsroot-1e-6-bs64

GPT-2 (124M) fine-tuned on EleutherAI/bergson-wikitext-512-chunks with the bergson MAGIC trainer, for use as a base model in training data attribution experiments.

Training details

• Optimizer: AdamW (beta1=0.95, beta2=0.975, eps_root=1e-6, weight_decay=0.01)
• Batch size: 64, epochs: 4, seed: 42, precision: fp32
• LR: polynomial schedule, peak 8e-4, warmup 25%, end 8e-5
• Trained with bergson 0.10.1 (git sha 27eb22b3)

Replication

The exact bergson run config is included as bergson_config.yaml. Replay it with:

bergson bergson_config.yaml

A companion model trained identically but with eps_root=0 is available at EleutherAI/gpt-2-wikitext-chunks-epsroot-0-bs64.