121–127 of 127 posts
I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
FWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).
Earlier quoted context omitted.
FWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).
How much % of the theoretical FLOPs are you getting with those 7900 XTX on training?
Earlier quoted context omitted.
Did you reproduce the evaluation as well?
So... successfully reproduced in ~8.75 hours, taking about 18 kWh / $2.70 The first run actually failed at step 3000 or so, and I realized I had a bug in my attention / matmul kernels, but after fixing that and restarting it worked great [1] https://github.com/anthonix/llm.c
Earlier quoted context omitted.
Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
Nah, I reproduced on 4x 7900 XTX machine in 8.75 hours, so a single 7900 XTX (costs less than $1k) could do it in under 24 hours. Was hitting 55.4% MFU.
Earlier quoted context omitted.
Spend one year to study multiple languages - bash, C, C++, Go, Python ... and even Mojo or Rust. 10-20 hours a week. Being able to read top programming languages is the best investment I ever made. You will become fearless and can see the matrix ;)
I did this and wrote about my experience: https://mode80.github.io/7-langs-in-12-months.html I don't regret it. But if ML is your main goal, Python is where you will end up because it's where the libraries are.
Earlier quoted context omitted.
So... successfully reproduced in ~8.75 hours, taking about 18 kWh / $2.70 The first run actually failed at step 3000 or so, and I realized I had a bug in my attention / matmul kernels, but after fixing that and restarting it worked great [1] https://github.com/anthonix/llm.c
What was the final loss? Is this hardware available for rent somewhere?
step 18865/18865 | train loss 3.280550 | norm 0.4362 | lr 0.00e+00 | 1669.06 ms | 55.4% A100 fp16 MFU | 314058 tok/s Writing state to log124M/state_00018865_00003.bin val loss 3.296179
You can buy these GPUs on Amazon for under $1k. I heard the MI300X may be available in Azure now or at least very soon.