Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CS336: Language Modeling from Scratch

cs336.stanford.edu

Post reply on HN
Print view View original

41–50 of 56 posts

Re: CS336: Language Modeling from Scratch

#41

Post by sonabinu »

armas wrote: post #29

I independently worked on the first two assignments over the course of a year. I learned so much! I was wondering what other courses people took on afterwards :)

I’m watching the Frontier Systems videos - https://cs153.stanford.edu/

The one issue I had with the CS336 course was the delivery of the RL components. I liked Lectures 5 & 6 from CME 295 better

https://cme295.stanford.edu/syllabus/

I’ve heard good things about the diffusion models class as well - CME 296. Seems like a good next step.

https://cme296.stanford.edu/syllabus/

Re: CS336: Language Modeling from Scratch

#42

Post by alok-g »

pell wrote: post #35
AJRF wrote: post #33

Can anyone answer question - whats the minimum viable GPU to follow along with this course at home? I have a 5080 16GB, are they really needing more than that in this course?

The first section can be done on a M1 chip, I think the second one needs Triton support, so your 5080 should be fine.

How about NVIDIA GeForce RTX 2060 (6 GB)? Would that be sufficient too? I am using Windows 11. Thanks.

Re: CS336: Language Modeling from Scratch

#43

Post by 5555watch »

marcelroed wrote: post #22
storus wrote: post #2

Thanks for releasing this again! What are this year's changes to prior offerings?

TA here. Biggest changes are in the second assignment (distributed) where we added a bunch of memory, profiling and distributed tasks, as well as in the fifth assignment (alignment), where most of the RL tasks are fresh this year. Assignment 3 (scaling laws) was also completely updated, but in a way that might be difficult to run without substantial resources. I'm working on a way for external students to be able to…

How are you grading the student submissions? Also, do you catch students who fully use AI and don't follow the Honor code? If so, how?

Re: CS336: Language Modeling from Scratch

#45

Post by tevlon »

Couple days ago, i used Claude to implement an improved version of gpt-1. I am no ML Engineer by no means. I am just a normal backend engineer. I ended up creating a hybrid between gpt-1 and modded-nanogpt (from KellerJordan).

I was able to reproduce the results of the original gpt-1 paper with my gaming PC. I don't even have alot of VRAM. My NVIDIA GeForce RTX 2060 SUPER was able to reproduce most of the results with just 1 hour of training. I would totally recommend to do the same, if you are interested in pre-training LLMs.

The code is here: https://github.com/epoyraz/modded-gpt-1 But, you can also just ask Claude 4.8 or Codex 5.5

Re: CS336: Language Modeling from Scratch

#46

Post by marcelroed »

5555watch wrote: post #43
marcelroed wrote: post #22

Earlier quoted context omitted.

TA here. Biggest changes are in the second assignment (distributed) where we added a bunch of memory, profiling and distributed tasks, as well as in the fifth assignment (alignment), where most of the RL tasks are fresh this year. Assignment 3 (scaling laws) was also completely updated, but in a way that might be difficult to run without substantial resources. I'm working on a way for external students to be able to…

How are you grading the student submissions? Also, do you catch students who fully use AI and don't follow the Honor code? If so, how?

We have autograding for code through tests written by hand, and additionally do manual code audits if we see suspicious behavior. We also do grading the old-fashioned way for writeups.

We do indeed catch students who don't follow the honor code. It's very obvious from how the code looks, as well as the rate of progress. Since we use Modal for class submissions, we have code deltas for every time they run something on B200s. The diffs often contain something like 300 lines in 5 minutes, in which case we review and report based on how egregious/provable it looks.

Re: CS336: Language Modeling from Scratch

#47

Post by [deleted] »

danbrooks wrote: post #16
airstrike wrote: post #10

I wonder if people prefer to learn this on their own or if building a community around open learning is something that others are interested in

I'd be interested in joining a discord server. Would be great to have a community to discuss the material - even if folks can't commit to the full course.

[deleted]

Re: CS336: Language Modeling from Scratch

#49

Post by jvican »

wandering-nomad wrote: post #48

Can the assignments be done on MBP M5 Max? I am hoping to get my hands on one in next couple of weeks and really want to pursue this course

Most of them can, but assignment 2 requires Nvidia GPUs as it heavily depends on you getting acquainted with Triton, Nsight compute and other GPU low-level programming tools.

Re: CS336: Language Modeling from Scratch

#50

Post by zaptrem »

marcelroed wrote: post #34
fg137 wrote: post #30

I recently completed the 2025 version of this course (video + most assignments, skipping some of the most costly part of the tasks). That's quite something. There is a lot going on in the first two assignments which required a ton of thinking and debugging. Despite having a decent foundation in deep learning, it took me several months to finish it using bits of my after-work hours and weekends. (I am not a model part…

TA here. Noted! I now have more resources to test more environments, and will do so whenever possible. I think freezing due to memory overuse is going to be a problem with anything you code yourself, but I do think we could be more rigorous with guiding people to achieve limited memory use for the tokenizer task. IMO the cost of renting GPUs is a bit overstated in these comments. Generally almost all of the developme…

OOM on CUDA GPUs is relatively graceful (the process crashes). However, on macOS if torch MPS tries to allocate too much memory, the whole kernel will simply lock up and the only option is to reboot the computer. I have no idea why Apple doesn’t reserve memory for stuff like the OOM/kernel watchdog, but it seems they either don’t or there is a bug.
Post reply on HN