I wish I knew how to do yolo runs.
- signed, a compute resource hog at FAANG
Training LLMs from ground zero as a startup
61–70 of 125 posts
Re: Training LLMs from ground zero as a startup
#62Earlier quoted context omitted.
Good point, so the only real differentiator would be the size & quality of the data being fed and the fine tuning done on the model? I wonder what else differentiates LLMs from each other
Alignment and censorship ?
Alignment just means making it do what you want. LLMs just continue the sequence, the chat questions and response style we have now is an example of alignment (to what humans want).
Re: Training LLMs from ground zero as a startup
#63Re: Training LLMs from ground zero as a startup
#64Earlier quoted context omitted.
To be honest, most researchers in applied ML in the bay say the opposite. If you are trying to be nimble and prototype, use pytorch. If you're trying to gain some optimizations as you near deployment, rewrite in Jax.
Where does Tensorflow stand in this?
Tensorflow has been falling behind since they stopped caring about backward compatibility. PyTorch is the leading framework. Jax is getting some traction at Google and was used to train Gemini.
Re: Training LLMs from ground zero as a startup
#65Earlier quoted context omitted.
To be honest, most researchers in applied ML in the bay say the opposite. If you are trying to be nimble and prototype, use pytorch. If you're trying to gain some optimizations as you near deployment, rewrite in Jax.
Where does Tensorflow stand in this?
[dead]
Re: Training LLMs from ground zero as a startup
#66I’m wondering if the title should read “from the ground up” instead of “ground zero”? https://en.wikipedia.org/wiki/Hypocenter
Re: Training LLMs from ground zero as a startup
#67Re: Training LLMs from ground zero as a startup
#68It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.
OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…
Ha! We’re also committing great sins of computation against T4s at our company. Hopefully, as I learn, things get less janky.
Re: Training LLMs from ground zero as a startup
#69It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.
OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…
> And we are using the older Tesla T4.
That's an undemanding and well-debugged chip by this point (6 years ago!). So you aren't experiencing any of the pain people using A100s or H100s (never mind people who have to stand up clusters with B100s soon) are going through now.
Re: Training LLMs from ground zero as a startup
#70Earlier quoted context omitted.
> it is so overcome by historical events that you will confuse your audience I really don't think so. I think almost every US president has used that exact phrase ("nuclear holocaust") in the last 20 years. Trump, Biden, Obama, Bush, have all casually used the phase "nuclear holocaust" while speaking to the American public. > they aren't wrong about the meaning of the phrase to contemporary english speakers. I am a c…
I was just trying to be considerate to the most recent and arguably quintessential use of the phrase, with regards to ground zero and 9/11. The point I was aiming at was the general semantic meaning of the phrase. It seemed mismatched from the meaning intended by the sentence. With startups, there are various phrases, like "get in on the ground floor/level" or "build it from the ground up" where the full semantic met…
I do agree with you that a different phase would have been better and also the author might have been mixing metaphors like you alluded to ("get in on the ground floor/level" or "build it from the ground up").
I didn't read the complete article so I missed the phrase originally, it might have given me a slight pause but I would understand the meaning.
My apologies too for the nitpicking, I enjoy language and yes, interesting were on an LLM thread discussing this.