Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Smol Training Playbook: The Secrets to Building World-Class LLMs

huggingface.co

Post reply on HN
Print view View original

11–20 of 20 posts

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#11

Post by potsandpans »

forgingahead wrote: post #8

Where does "Smol" come from? It's supposed to mean "Small" right? If yes then what's the etymology and reason for popular usage?

It's just internet speak from the days of tumbler. It usually has cutsie connotations.

Tumbler speak has a bunch of whacky things, notably "chimkin nuggers."

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#12

Post by lewtun »

forgingahead wrote: post #8

Where does "Smol" come from? It's supposed to mean "Small" right? If yes then what's the etymology and reason for popular usage?

In the specific case of SmolLM, it originates from the meme in this dataset https://huggingface.co/datasets/bigcode/the-stack-smol

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#13

Post by yorwba »

CamperBob2 wrote: post #10
doctorpangloss wrote: post #9

I really like the Hugging Face guys, but... > Modify one thing at a time > Change only one variable per ablation while keeping everything else constant. If you change multiple things and performance improves, you won’t know what caused it. Test modifications individually, then combine successful ones and reassess. This is an unintentional microcosm of what is flawed with the document.

What's wrong with it? That's good advice in almost any optimization or troubleshooting context where variables may interact.

One problem with testing one change at a time is that if you can only run a small number of experiments because each one requires many GPU hours to get results, you can also only test a small number of changes. If you can come up with and implement new changes much more easily than you can test them, it would be more efficient to test multiple changes at a time and use some form of Bayesian optimization to find the best combination of changes with as few experiments as possible.

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#14

Post by ImageXav »

yorwba wrote: post #13
CamperBob2 wrote: post #10

Earlier quoted context omitted.

What's wrong with it? That's good advice in almost any optimization or troubleshooting context where variables may interact.

One problem with testing one change at a time is that if you can only run a small number of experiments because each one requires many GPU hours to get results, you can also only test a small number of changes. If you can come up with and implement new changes much more easily than you can test them, it would be more efficient to test multiple changes at a time and use some form of Bayesian optimization to find the b…

Agreed. One at a time testing (OAT) has been outdated for almost a century at this point. Factorial and fractional factorial experiments have been around for that long and give detailed insights into the effect of not just single changes but the interaction between changes, which means you can superpower your learnings as many variables in DL do in fact interact.

Or, more modern Bayesian methods if you're more interested in getting the best results for a given hyperparameter sweep.

However, that is not to detract from the excellent effort made here and the great science being investigated. Write ups like this offer so much gold to the community.

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#15

Post by matusp »

lewtun wrote: post #5

Hi, Lewis here (one of the co-authors). Happy to answer any questions people have about the book :)

Really impressive writeup. In your opinion, how long will this stay up to date? The field is constantly evolving, do you plan to keep updating this document?

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#16

Post by lewtun »

matusp wrote: post #15
lewtun wrote: post #5

Hi, Lewis here (one of the co-authors). Happy to answer any questions people have about the book :)

Really impressive writeup. In your opinion, how long will this stay up to date? The field is constantly evolving, do you plan to keep updating this document?

Thanks! I expect the book will remain relevant as long as the Transformers architecture does. That’s why we mostly focus on topics we think will stand the test of time, but let’s see how that plays out :)

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#17

Post by matusp »

yorwba wrote: post #13
CamperBob2 wrote: post #10

Earlier quoted context omitted.

What's wrong with it? That's good advice in almost any optimization or troubleshooting context where variables may interact.

One problem with testing one change at a time is that if you can only run a small number of experiments because each one requires many GPU hours to get results, you can also only test a small number of changes. If you can come up with and implement new changes much more easily than you can test them, it would be more efficient to test multiple changes at a time and use some form of Bayesian optimization to find the b…

The number of runs you can afford are not enough to perform Bayesian optimization. Count how many different options they explored in the text and take a guess how many samples you need to start modeling the hyperparameter space.

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#18

Post by doctorpangloss »

CamperBob2 wrote: post #10
doctorpangloss wrote: post #9

I really like the Hugging Face guys, but... > Modify one thing at a time > Change only one variable per ablation while keeping everything else constant. If you change multiple things and performance improves, you won’t know what caused it. Test modifications individually, then combine successful ones and reassess. This is an unintentional microcosm of what is flawed with the document.

What's wrong with it? That's good advice in almost any optimization or troubleshooting context where variables may interact.

It’s advice for being an individual contributor, not a researcher.

And even then. If you’re an IC and your boss is saying, “incrementalism at the level of planning experiments,” and the goal is research, quit, because you will fail.

Re: The Smol Training Playbook: The Secrets to Building World-Class LLMs

#19

Post by troelsSteegin »

lewtun wrote: post #5

Hi, Lewis here (one of the co-authors). Happy to answer any questions people have about the book :)

This was a good read. I was struck by the quantity of nuanced and applied knowhow it took to build SmolLM3. I am curious about the rough cost it took to engineer and train SmolLM3 - at ~400 GPUS for a least a month, and, based on the set of book co-authors, 12 engineers for at least three months. Is $3-5M a fair ballpark number? The complement is how much experience, on average, the team members had doing ML and LLM training at scale before SmolLM3. The book is "up" on recent research, so I am surmising a phd-centric team each with multiple systems built. This is not commodity skill. What the book suggests to me is that an LLM applications start up would best focus on understanding the scope and knowhow for starting from post-training.
Post reply on HN