Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Training LLMs from ground zero as a startup

#111

Post by flybarrel »

abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

Newbie question - What happens after when an LLM training job experience a hardware failure? I don't suppose you lose all the training progress do you? Then the pain is mostly in the diagnostic of the problem and getting the cluster running again, but no need to worry about data loss right?

Re: Training LLMs from ground zero as a startup

#112

Post by ein0p »

alsoworkedthere wrote: post #101
ein0p wrote: post #87

Earlier quoted context omitted.

A recent ex-googler here: quality of Google3 in general is pretty good, but the LLM training bits are so abysmal that I know people who have resigned instead of working on it. And it’s also extra slow because getting a couple local GPUs is not really an option. So you’re forced to “develop in Colab” which works for some things and not for others and in general sucks ass if you’re working on anything substantial. For…

Finally, an accurate portrayal! Google has superb robustness and code quality, with garbage-level usability. Once you're setup, you can kick off many massive training jobs and compare results easily. However, getting to that point is really hard. You'll never figure out how to use the ML infrastructure and libraries on your own. You can only get it to work by meeting with the teams that wrote the infra so they can fi…

Google doesn’t use VMs internally to run workloads. But yeah, seconds-long dev iteration cycles take minutes or even tens of minutes there.

Re: Training LLMs from ground zero as a startup

#113

Post by fennecbutt »

llm_trw wrote: post #59
joe_the_user wrote: post #6

So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these natur…

>Yet overall, this seems like a massive, energy intensive exercise in redundancy. Keep in mind that this is also chaff to distract people from the real secret sauce. I imagine that just as many startups are hiring writers and photographers to create extremely well labelled uncontaminated data for training. One only need to look at the perverts over at civitai to see how far you can go with intensive labeling on a tin…

Us furries were properly tagging data on e6 for a long time before LLMs came about.

Re: Training LLMs from ground zero as a startup

#114

Post by PeterStuer »

polygamous_bat wrote: post #108
PeterStuer wrote: post #91

Earlier quoted context omitted.

"this seems like a massive, energy intensive exercise in redundancy" This is commonly refered to as a market working as intended. Yes, the waste from this type of redundency can be massive , especially if you realize that ultimately just a tiny percentage of these efforts will result in even moderate success. But it is the price to pay at the edge of progress. A planned monopoly might be more efficient (despite popul…

> This is commonly refered to as a market working as intended. Is it? Seems like market is unable to separate wheat from the chaff and is just throwing money around hoping to hit the jackpot. While AI has massive chance of affecting our lives, the investment market paints a pretty similar picture to what happened during the crypto boom.

Our inability to predict future success from failiure is exactly why we have (massively inefficient) markets outcompeting centralized planned approaches.

Re: Training LLMs from ground zero as a startup

#115

Post by manquer »

polygamous_bat wrote: post #108
PeterStuer wrote: post #91

Earlier quoted context omitted.

"this seems like a massive, energy intensive exercise in redundancy" This is commonly refered to as a market working as intended. Yes, the waste from this type of redundency can be massive , especially if you realize that ultimately just a tiny percentage of these efforts will result in even moderate success. But it is the price to pay at the edge of progress. A planned monopoly might be more efficient (despite popul…

> This is commonly refered to as a market working as intended. Is it? Seems like market is unable to separate wheat from the chaff and is just throwing money around hoping to hit the jackpot. While AI has massive chance of affecting our lives, the investment market paints a pretty similar picture to what happened during the crypto boom.

is it any different from evolution?

Re: Training LLMs from ground zero as a startup

#116

Post by shrubble »

flybarrel wrote: post #110
shrubble wrote: post #88

Earlier quoted context omitted.

Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...

What would be an ideal (or more appropriate) speed?

Well it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.

Re: Training LLMs from ground zero as a startup

#117

Post by mc10 »

flybarrel wrote: post #110
shrubble wrote: post #88

Earlier quoted context omitted.

Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...

What would be an ideal (or more appropriate) speed?

simdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson

Re: Training LLMs from ground zero as a startup

#118

Post by nlpnerd »

TrackerFF wrote: post #92

Big question is, how do small startups manage to get funding for LLM products if they don’t have the “correct” background / pedigree? The world of LLM startups is beginning to look like the world of hedge funds and private equity firms - where the prerequisites for seed/funding are: A) Prestigious employment history / correct pedigree. B) Solid network of investors ready to jump before any product has even begun.

They don't. This is probably one reason why VCs invest in these companies. There is a natural moat since there is only a very finite number of people in the world has the right experience to raise, and only those who can raise can ever have the experience.

At least until compute cost drop to a cheap enough level...

Re: Training LLMs from ground zero as a startup

#119

Post by friendzis »

eru wrote: post #107
friendzis wrote: post #95

Earlier quoted context omitted.

In what way does this do more good than harm?

In the sense of people caring about their models not saying embarrassing things? Different people have different goals, and they don't necessarily align with yours.

Since the entity releasing the model obviously has certain goals aligning/censoring model in some ways is good for their particular short-term goal.

In the grand scheme these alignments are harmful as they place a reality distortion field. Authors create model of what language is and then contort that model to fit an opinionated idea of what language should be. Smells a bit Orwellian, right?

Re: Training LLMs from ground zero as a startup

#120

Post by eru »

friendzis wrote: post #119
eru wrote: post #107

Earlier quoted context omitted.

In the sense of people caring about their models not saying embarrassing things? Different people have different goals, and they don't necessarily align with yours.

Since the entity releasing the model obviously has certain goals aligning/censoring model in some ways is good for their particular short-term goal. In the grand scheme these alignments are harmful as they place a reality distortion field. Authors create model of what language is and then contort that model to fit an opinionated idea of what language should be. Smells a bit Orwellian, right?

> Smells a bit Orwellian, right?

No, seems perfectly fine by me. You are already shaping your results by your selection of training data. Eg do you want to train a model that speaks English, or German, or both? Do you want to run your training data past a spam filter first? Do you want to do a character based model, or one of those weird encodings that is popular with LLMs these days?

Doing some other procedures afterwards to make sure your LLM doesn't say embarrassing things is small fries by comparison.

Also it's good practice for trying to get alignment with more important values (like "don't kill all humans") later when models might get powerful enough to be able to kill all humans.

Playing some little games where OpenAI tries to keep you from making their model say embarrassing things, and people keep trying to make it say embarrassing things, is a good low stakes practice ground.

Post reply on HN