Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

Post reply on HN
Print view View original

81–90 of 125 posts

Re: Training LLMs from ground zero as a startup

#81

Post by bigcat12345678 »

3abiton wrote: post #76
swyx wrote: post #2

for context Yi Tay was Tech Lead on Google PaLM, UL2, Flan, Bard, etc and now is cofoudner at Reka (which has shipped some v interesting small multimodal models that have featured on here). I prompted him for this post as an ex-Googler now training LLMs as an independent startup https://twitter.com/YiTayML/status/1765105066263052718 our conversation was recorded here https://sub.thursdai.news/p/thursdai-feb-15-2024-o…

Is he the person after the Yi LLM model?

No Yi LLM models are from [0], Kaifu Li's LLM startup.

[0] https://www.lingyiwanwu.com/

Re: Training LLMs from ground zero as a startup

#82

Post by jen20 »

anonymous-panda wrote: post #20
jen20 wrote: post #16

Earlier quoted context omitted.

I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.

Packaging and stability reasons. Same for why it’s a 1.7gb install - probably where they landed after having tons of support issues on some random Python version they didn’t test or some issue with a dependency that had that issue. Freezing the entire set of artifacts is more stable and Python lets you move pretty quick. I can’t speak to why nodejs vs Python though - maybe Python is easier to embed?

Of course, writing them in Go would solve all of these problems while producing packages which are much smaller.

Re: Training LLMs from ground zero as a startup

#85

Post by eru »

pests wrote: post #62
Iulioh wrote: post #54

Earlier quoted context omitted.

Alignment and censorship ?

Alignment just means making it do what you want. LLMs just continue the sequence, the chat questions and response style we have now is an example of alignment (to what humans want).

Alignment can mean making sure your LLM doesn't continue the sequence in embarrassing ways, eg by spouting politically incorrect sequences of words (even though those might have been common in the training data).

Re: Training LLMs from ground zero as a startup

#86

Post by polygamous_bat »

a_bonobo wrote: post #83

But what is the product they're selling? The main Reka.AI page looks like a regular ChatGPT clone, an LLM you pay for by the token. How is this different from all these other companies? Pricing seems to be comparable to ChatGPT 3.5-Turbo.

Perhaps a cure for venture capitalist FOMO for not having invested in AI?

Re: Training LLMs from ground zero as a startup

#87

Post by ein0p »

twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

A recent ex-googler here: quality of Google3 in general is pretty good, but the LLM training bits are so abysmal that I know people who have resigned instead of working on it. And it’s also extra slow because getting a couple local GPUs is not really an option. So you’re forced to “develop in Colab” which works for some things and not for others and in general sucks ass if you’re working on anything substantial. For anything more substantial you’ll be launching stuff on some resource pool, waiting for like 10-15 minutes until it starts (much longer for large models), and then trying to divine why it failed from voluminous and sometimes indecipherable crash logs which also hang your browser when cluster UI tries to load them.

Rumors of Google’s AI code superiority are vastly overblown in 2024. I’m currently at another major AI lab, and the code here can actually be understood and worked on, which I consider to be a massive advantage.

Re: Training LLMs from ground zero as a startup

#88

Post by shrubble »

ganeshkrishnan wrote: post #52
abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…

Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...

Re: Training LLMs from ground zero as a startup

#89

Post by [deleted] »

ganeshkrishnan wrote: post #52
abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…

[deleted]

Re: Training LLMs from ground zero as a startup

#90

Post by pama »

lyapunova wrote: post #42
axpy906 wrote: post #23

Earlier quoted context omitted.

If you’re doing research JAX makes some sense. Probably some Google bias in there too.

To be honest, most researchers in applied ML in the bay say the opposite. If you are trying to be nimble and prototype, use pytorch. If you're trying to gain some optimizations as you near deployment, rewrite in Jax.

Interesting perspective about possible Jax optimizations. Assuming these models are trained and deployed on non-TPU hardware, are there any real advantages in using Jax for deployment on GPU? I’d have assumed that inference is largely a solved optimization for large transformer based models (with any low hanging fruits from custom CUDA code already written) and the details are shifting towards infrastructure tradeoffs and availability of efficient GPUs. But I may be out of the loop with the latest gossip. Or do you simply mean that maybe there exist cases where TPU inference makes sense financially and using jax makes a difference?
Post reply on HN