Qwen/Qwen3.8-27B
Image-Text-to-Text • 28B • Updated • 5.74M • • 14k
Nick B PRO
drrockso72
·
AI & ML interests
None yet
Recent Activity
liked a model 22 days ago
Qwen/Qwen3.8-27B liked a model 3 months ago
sapientinc/HRM-Text-1BOrganizations
None yet
liked a
model 22 days ago
reacted to eabdullin's post with 👍 3 months ago
Post
5752
I’m doing a PhD in AI, which sounds impressive until you realize it mostly means I spend three years trying to make a computer say something slightly less stupid than it said yesterday.
People hear "AI researcher" and they think I’m building the future. No. I’m in a basement at 2 a.m. Googling, "CUDA error what the f**k does this mean."
And the worst part about AI research now is compute. You don’t even ask, "Is this idea good?" anymore. You ask, "Can I afford for this idea to be wrong?"
My advisor comes to me one day and says, "I think we should fine-tune our own language model."
I said, "Professor, with what money? I’m a PhD student. I have two bank accounts: checking and emotionally checking."
He goes, "Don’t worry. We have compute."
Now, in academia, "don’t worry" is never the beginning of a good sentence.
I said, "What do you mean we have compute?"
He said, "My friend knows the cluster admin. He can get us on the GPUs."
I said, "Okay… what do we have to do?"
He goes, "Nothing crazy. Just be very grateful in the acknowledgements."
I said, "How grateful?"
He said, "Maybe put him as co-author."
I said, "Co-author? Are we using the cluster, or is the cluster using us?"
Because at that point, that’s not a favor. That’s academic child support.
So I go to the server room, and the cluster admin walks up to me and goes, "So you’re the NLP student."
And in my head I’m like, "No, tonight you’re the principal investigator. You’re the provider. I’m just a little token waiting to be attended to."
Because whoever controls the GPUs controls the relationship. That’s lab romance.
He starts setting things up, and I’m trying to act casual, but I don’t understand any of the numbers he’s saying.
He’s like, "Yeah, I can probably give you four H100s for the weekend."
I’m nodding like, "Mmm. Four. Weekend. H. One hundred. Absolutely."
Inside I’m like, "Is that good? Is that prison time? Why did he say it like he was offering me organs?"
[Continue in comments...]
People hear "AI researcher" and they think I’m building the future. No. I’m in a basement at 2 a.m. Googling, "CUDA error what the f**k does this mean."
And the worst part about AI research now is compute. You don’t even ask, "Is this idea good?" anymore. You ask, "Can I afford for this idea to be wrong?"
My advisor comes to me one day and says, "I think we should fine-tune our own language model."
I said, "Professor, with what money? I’m a PhD student. I have two bank accounts: checking and emotionally checking."
He goes, "Don’t worry. We have compute."
Now, in academia, "don’t worry" is never the beginning of a good sentence.
I said, "What do you mean we have compute?"
He said, "My friend knows the cluster admin. He can get us on the GPUs."
I said, "Okay… what do we have to do?"
He goes, "Nothing crazy. Just be very grateful in the acknowledgements."
I said, "How grateful?"
He said, "Maybe put him as co-author."
I said, "Co-author? Are we using the cluster, or is the cluster using us?"
Because at that point, that’s not a favor. That’s academic child support.
So I go to the server room, and the cluster admin walks up to me and goes, "So you’re the NLP student."
And in my head I’m like, "No, tonight you’re the principal investigator. You’re the provider. I’m just a little token waiting to be attended to."
Because whoever controls the GPUs controls the relationship. That’s lab romance.
He starts setting things up, and I’m trying to act casual, but I don’t understand any of the numbers he’s saying.
He’s like, "Yeah, I can probably give you four H100s for the weekend."
I’m nodding like, "Mmm. Four. Weekend. H. One hundred. Absolutely."
Inside I’m like, "Is that good? Is that prison time? Why did he say it like he was offering me organs?"
[Continue in comments...]
- 1 reply
liked a
model 3 months ago
liked a
dataset 5 months ago
liked a
dataset 6 months ago
upvoted an article 6 months ago
Article
Scaling Pedagogical Pre-training: From Optimal Mixing to 10 Billion Tokens
codelion
• • 6As an independent researcher who creates models ranging from 30m to 2.5b params on a single 5090 in his living room, I find your work invaluable. Thank you!
reacted to codelion's post with 🔥 6 months ago
Post
3448
Scaling Pedagogical Pre-training to 10 Billion Tokens
New blog post exploring what happens when you take optimal data mixing insights and scale up the data generation itself.
We built Sutra, a multi-stage framework for generating pedagogical pre-training data guided by a knowledge graph of ~2,000 concepts across 9 domains. The pipeline includes structured content generation, six-dimension quality evaluation, diversity management across 20 content styles, and a cleaning stage to prevent collapse.
The result is codelion/sutra-10B, a 10.2 billion token pedagogical dataset with rich metadata (domain, complexity, prerequisites, quality scores) on every entry.
We trained codelion/SmolLM2-70M on it for 3 full epochs (30.6B tokens) on a single A10 GPU in ~78 hours.
Key finding: perplexity kept improving across epochs, but benchmark gains plateaued fast. At 70M parameters, the model hits a representational ceiling that more data alone can't break through.
Full writeup with comparisons against 7 other datasets, detailed benchmark breakdowns, and connections to recent work on synthetic data scaling, curriculum learning, and data mixing laws: https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens
All datasets at multiple scales (10M, 100M, 1B, 10B) plus seed concepts and an SFT variant are in the Sutra Pedagogical Datasets collection.
New blog post exploring what happens when you take optimal data mixing insights and scale up the data generation itself.
We built Sutra, a multi-stage framework for generating pedagogical pre-training data guided by a knowledge graph of ~2,000 concepts across 9 domains. The pipeline includes structured content generation, six-dimension quality evaluation, diversity management across 20 content styles, and a cleaning stage to prevent collapse.
The result is codelion/sutra-10B, a 10.2 billion token pedagogical dataset with rich metadata (domain, complexity, prerequisites, quality scores) on every entry.
We trained codelion/SmolLM2-70M on it for 3 full epochs (30.6B tokens) on a single A10 GPU in ~78 hours.
Key finding: perplexity kept improving across epochs, but benchmark gains plateaued fast. At 70M parameters, the model hits a representational ceiling that more data alone can't break through.
Full writeup with comparisons against 7 other datasets, detailed benchmark breakdowns, and connections to recent work on synthetic data scaling, curriculum learning, and data mixing laws: https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens
All datasets at multiple scales (10M, 100M, 1B, 10B) plus seed concepts and an SFT variant are in the Sutra Pedagogical Datasets collection.
- 2 replies
Not available for individuals?
👍 11
5
#2 opened 9 months ago
by 
blythet