Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

Post reply on HN
Print view View original

101–110 of 125 posts

Re: Training LLMs from ground zero as a startup

#101

Post by alsoworkedthere »

ein0p wrote: post #87
twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

A recent ex-googler here: quality of Google3 in general is pretty good, but the LLM training bits are so abysmal that I know people who have resigned instead of working on it. And it’s also extra slow because getting a couple local GPUs is not really an option. So you’re forced to “develop in Colab” which works for some things and not for others and in general sucks ass if you’re working on anything substantial. For…

Finally, an accurate portrayal!

Google has superb robustness and code quality, with garbage-level usability. Once you're setup, you can kick off many massive training jobs and compare results easily. However, getting to that point is really hard. You'll never figure out how to use the ML infrastructure and libraries on your own. You can only get it to work by meeting with the teams that wrote the infra so they can find and fix every error and misconfiguration. Usually, there is one single way to get things working together, and neither the documentation nor the error messages will get you to that brittle state.

It's near impossible to get a VM with a TPU or GPU attached, so there's no way to debug issues that happen between the library and the accelerator. Plus somehow they've made Python take longer to build (??!!) and run than C++ takes, so your iteration cycle is several minutes for what would take seconds at any other place. Fun stuff! Somehow it's still one of the best places to do ML work, but they sure try to make it as difficult as possible.

Re: Training LLMs from ground zero as a startup

#102

Post by frozenseven »

dotancohen wrote: post #98
planet_y wrote: post #66

I’m wondering if the title should read “from the ground up” instead of “ground zero”? https://en.wikipedia.org/wiki/Hypocenter

Yes, the title sounds like somebody confused two idioms. That's not the type of author from whom I want to learn.

1. As others have pointed out, it's a perfectly valid idiom. Check a dictionary.

2. How do you think idioms are created in the first place?

3. What exactly forces you to act like this?

Re: Training LLMs from ground zero as a startup

#103

Post by jen20 »

anonymous-panda wrote: post #37
pests wrote: post #24

Earlier quoted context omitted.

What? They only get package and stability because they include the runtime. If they just went with a compiled language they could distribute native binaries and have actual packaging and stability.

Yes, but it’s not just a single metric. Another is how easy it is for them to hire productive members of the team and how much that costs them - middling Python developers churning out fine”ish” code are cheaper than Rust developers doing the same. It’s hard to find a language where you can be as productive as a developer in Python that also has AOT compilation to generate standalone binaries. Tldr: there’s multiple…

> It’s hard to find a language where you can be as productive as a developer in Python that also has AOT compilation to generate standalone binaries.

Outside specific cases around machine learning, it’s really not: Go is that language. It’s not like each of those platforms doesn’t have to have a similar team that understand Go anyway (for their SDK), so they could save their customers the abject pain of Python dependency management by just writing their CLIs using it.

Re: Training LLMs from ground zero as a startup

#104

Post by tkgally »

I learned about reka.ai from this post; their LLMs don’t seem to have been discussed much on HN yet [1]. So, out of curiosity, I spent the last hour testing prompts with their chat interface [2] in comparison with ChatGPT 4, Gemini Advanced, Claude 3, and Mistral Large. I put the results at [3]. Overall, Reka Flash doesn’t seem significantly worse or better than the others. A lot more testing would be necessary to be sure, of course.

[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

[2] https://chat.reka.ai/chat

[3] https://gally.net/temp/20240307llmcomparison.html

Re: Training LLMs from ground zero as a startup

#105

Post by nl »

ganeshkrishnan wrote: post #52
abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…

> 20GB json file… takes 6 hours just to load

Err you definitely should be doing something about that.

20GB on T4s (how many?) isn’t really comparable to terabytes on thousands of A100s.

Re: Training LLMs from ground zero as a startup

#106

Post by egberts1 »

TL:DR: LLM training is highly susceptible to GIGO.

(GIGO is what one gets when feeding LLM with "G"arbage "I"n, "G"arbage "O"ut.)

This is the current problem about making a vaccine signature fitting like a glove ... as tight as possible ... when populating the anti-malware (i.e. IDS/IPS/NDS/XNS) search pattern engine for use by Aho-Corasick-variant algorithms (such as Parallel-Failureless Aho Corasick).

However, LLM as a binary code-based detector for malware detection has a very limited benefit (it is there but only as a backend topical add-on after all other conditionals have been identified).

LLM lacks qualifying conditionals surrounding a premise data, and I have my doubts of using LLM for medical diagnosis as well: until we start having LLM denote the much-needed weighted combo-conditionals by "percentages".

Re: Training LLMs from ground zero as a startup

#107

Post by eru »

friendzis wrote: post #95
eru wrote: post #85

Earlier quoted context omitted.

Alignment can mean making sure your LLM doesn't continue the sequence in embarrassing ways, eg by spouting politically incorrect sequences of words (even though those might have been common in the training data).

In what way does this do more good than harm?

In the sense of people caring about their models not saying embarrassing things?

Different people have different goals, and they don't necessarily align with yours.

Re: Training LLMs from ground zero as a startup

#108

Post by polygamous_bat »

PeterStuer wrote: post #91
joe_the_user wrote: post #6

So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these natur…

"this seems like a massive, energy intensive exercise in redundancy" This is commonly refered to as a market working as intended. Yes, the waste from this type of redundency can be massive , especially if you realize that ultimately just a tiny percentage of these efforts will result in even moderate success. But it is the price to pay at the edge of progress. A planned monopoly might be more efficient (despite popul…

> This is commonly refered to as a market working as intended.

Is it? Seems like market is unable to separate wheat from the chaff and is just throwing money around hoping to hit the jackpot. While AI has massive chance of affecting our lives, the investment market paints a pretty similar picture to what happened during the crypto boom.

Re: Training LLMs from ground zero as a startup

#109

Post by lambersley »

abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

Agreed. It reads like Seven of Nine realizing she's separated from the Collective and needs to rely lowly human capabilities. The insights into vendors was informative.

Re: Training LLMs from ground zero as a startup

#110

Post by flybarrel »

shrubble wrote: post #88
ganeshkrishnan wrote: post #52

Earlier quoted context omitted.

OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…

Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...

What would be an ideal (or more appropriate) speed?
Post reply on HN