Training LLMs from ground zero as a startup
1–10 of 125 posts
Re: Training LLMs from ground zero as a startup
#2for context Yi Tay was Tech Lead on Google PaLM, UL2, Flan, Bard, etc and now is cofoudner at Reka (which has shipped some v interesting small multimodal models that have featured on here). I prompted him for this post as an ex-Googler now training LLMs as an independent startup https://twitter.com/YiTayML/status/1765105066263052718
our conversation was recorded here https://sub.thursdai.news/p/thursdai-feb-15-2024-openai-chan...
Re: Training LLMs from ground zero as a startup
#3Training LLM from scratch is a super important issue that affects the pace and breadth of iteration of AI almost as much as the raw hardware improvements do. The blog is fun but somewhat shallow and not technical or very surprising if you’ve worked with clusters of GPUs in any capacity over the years. (I liked the perspective of a former googler, but I’m not sure why past colleagues would recommend Jax over pytorch for LLMs outside of Google.) I hope this newco eventually releases a more technical report about their training adventures, like the PDF file here: https://github.com/facebookresearch/metaseq/tree/main/projec...
Re: Training LLMs from ground zero as a startup
#4It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google".
This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.
Re: Training LLMs from ground zero as a startup
#5> All in all, this is only a small part of the story of how we started a company, raised some money, bought some chips and matched Gemini pro/GPT 3.5 and outperformed many others in less than a year having to build everything from scratch.
I wonder what was the budget spent for the chips/cloud GPUs to achieve GPT 3.5 level LLM - at least in the order to magnitude - 2-5 millions?
Re: Training LLMs from ground zero as a startup
#6So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms).
So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these naturally has their own brand of "secret sauce" for distinguishing their venture. The various secret sauces can make a difference in the quality of an LLM's output.
Yet overall, this seems like a massive, energy intensive exercise in redundancy.
Re: Training LLMs from ground zero as a startup
#7So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these natur…
I don't think most of them have any kind of secret sauce. I think the founders hope to get bought out simply for being able to train "near-SOTA" LLMs. I guess achieving that level of skill and infra could be valuable enough to build upon.
Re: Training LLMs from ground zero as a startup
#8[flagged]
Re: Training LLMs from ground zero as a startup
#9[flagged]
Well, GPU failure modes (from what Ive heard from ML infra people) are often subtle, eg incorrect multiplication results. So its not as simple as the usual ‘treat resources like cattle not pets’ because tou dont know which cows have mad cow disease before committing to an expensive training run
Re: Training LLMs from ground zero as a startup
#10[flagged]
I think it is a bit more nuanced than that. The author is experienced.