Runtime error
Agents
AIDC AI Marco O1
💻
Generate responses to text prompts for AI chat
alkinun
AtAndDev
AI & ML interests
decentralize
Recent Activity
liked a dataset 2 days ago
common-pile/project_gutenberg_filtered liked a dataset 3 days ago
IFM/Pretrain-Behaviors reacted to juiceb0xc0de's post with 🔥 3 days ago
Dude, Where's My Update? I'll tell you where! ~97.6% of my BF16 parameter coordinates didn't move at all, and the ones that did overshot by ~1.33x.
It's nice to do research that doesn't end in disproving yourself once again and moving on to the next subject once in awhile.
Back to the topic, if you've ever wondered why most of your weights are basically ghosting you nearly every step when you store your weights at bf16, Dude, I Measured It.
https://huggingface.co/blog/juiceb0xc0de/intended-and-realized-updates-in-bf16-fine-tuning#dude-wheres-my-updateOrganizations
