Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

Post reply on HN
Print view View original

41–50 of 125 posts

Re: Training LLMs from ground zero as a startup

#41

Post by observationist »

pests wrote: post #35
nyokodo wrote: post #33

Earlier quoted context omitted.

> We use it in phases like "a nuclear holocaust" while at the same time it refers to a specific horrible incident. The historical and still sometimes contemporary meaning of "holocaust" is a burnt offering at an altar. But, if you're using that term now in a general context to mean merely a burnt offering it is so overcome by historical events that you will confuse your audience, cloud out your message, and probably…

> it is so overcome by historical events that you will confuse your audience I really don't think so. I think almost every US president has used that exact phrase ("nuclear holocaust") in the last 20 years. Trump, Biden, Obama, Bush, have all casually used the phase "nuclear holocaust" while speaking to the American public. > they aren't wrong about the meaning of the phrase to contemporary english speakers. I am a c…

I was just trying to be considerate to the most recent and arguably quintessential use of the phrase, with regards to ground zero and 9/11.

The point I was aiming at was the general semantic meaning of the phrase. It seemed mismatched from the meaning intended by the sentence. With startups, there are various phrases, like "get in on the ground floor/level" or "build it from the ground up" where the full semantic metaphorical meaning of the phrase maps to the meaning intended. If you said you wanted to get in at ground zero, there's a semantic mismatch, so the meaning doesn't fully apply in the context in which it's used.

I don't think it's necessarily controversial, either. Comedians use the term "bombing" for performances gone bad. If a night went particularly bad, they could call it a holocaust. They also refer to "murdering" a crowd, when sets go really well - "my bit last night was a nuclear holocaust" might work to convey great success.

I gauge the level of correctness to the various levels of meaning and metaphor, so if multiple levels don't track, or if a singular level is mismatched to the context, then I don't consider it as a very correct use of the word or phrase. It's not totally wrong, but of the many ways in which that phrase is used correctly, it's not very right, either.

Anyway - My mistake for highlighting an admittedly minor issue among language model enthusiasts. We seem to enjoy delving into language's technicalities, and LLMs are great for exploring how meanings align with a sentence's intended message. I get caught up in these things easily.

Re: Training LLMs from ground zero as a startup

#42

Post by lyapunova »

axpy906 wrote: post #23
pama wrote: post #3

Training LLM from scratch is a super important issue that affects the pace and breadth of iteration of AI almost as much as the raw hardware improvements do. The blog is fun but somewhat shallow and not technical or very surprising if you’ve worked with clusters of GPUs in any capacity over the years. (I liked the perspective of a former googler, but I’m not sure why past colleagues would recommend Jax over pytorch f…

If you’re doing research JAX makes some sense. Probably some Google bias in there too.

To be honest, most researchers in applied ML in the bay say the opposite. If you are trying to be nimble and prototype, use pytorch. If you're trying to gain some optimizations as you near deployment, rewrite in Jax.

Re: Training LLMs from ground zero as a startup

#43

Post by danans »

twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

> Haven't worked at Google, anyone else share this sentiment?

I worked there, and the quality is definitely much higher and the code tends to be far more maintainable. However, there is often a cost for that, which is velocity.

Some of this is reduced by the sheer amount of automation in tooling (i.e. bots that block style violations and common bugs before a code change is submitted).

In other cases, it slows things down quite a bit.

Re: Training LLMs from ground zero as a startup

#45

Post by candiodari »

twelfthnight wrote: post #22
jen20 wrote: post #16

Earlier quoted context omitted.

I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.

There probably is a sense in which the API's are constantly changing, so maybe an interpreted language might make sense? I imagine there has to be a better way to do with with Go or Rust though (even lua?) for a smaller binary.

Google python binaries are more akin to docker or even vm images, even if the actual technology used predates docker and even linux VMs. They contain something like a slimmed-down linux distribution, not just a binary.

EXTREME predictability (e.g. as never ever using the system's libssl), in trade for huge binaries. They go pretty damn far in this: you won't catch a Google binary even using most of libc.

Re: Training LLMs from ground zero as a startup

#46

Post by joe_the_user »

abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

I took the phrase to mean "outside any large company". It seems like a fairly obvious metaphor; if you have a starup working on a large scale infrastructure project, you have to set your own logistics just a camp in the literal wildness.

Re: Training LLMs from ground zero as a startup

#47

Post by choppaface »

abeppu wrote: post #4

It's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.

Really telling quote:

> I was completely taken aback by the failure rate of GPUs as opposed to my experiences on TPUs at Google

Should be "I was completely unaware of the failure modes of GPUs, because all my career I've been inside Google and used Google TPUs and was well-acquainted with those failure modes."

I've used GPUs mostly, and when I tried TPUs the jobs failed all the time for really hard-to-debug reasons. Often the indirection between the x86 chip and the TPU device caused hours of hair-pulling, stuff you never get with x86+nvidia+pytorch.

10-15 years ago, Google minted many $10m+ data scientists (aka Sawzall engineers) who also ventured "into the wilderness" and had very similar reactions. This blog post is much more about the OP hyping his company and personal brand than contributing useful notes to the community.

Re: Training LLMs from ground zero as a startup

#49

Post by DeepChill »

joe_the_user wrote: post #27
dauertewigkeit wrote: post #7

Earlier quoted context omitted.

I don't think most of them have any kind of secret sauce. I think the founders hope to get bought out simply for being able to train "near-SOTA" LLMs. I guess achieving that level of skill and infra could be valuable enough to build upon.

Sure, that's also a factor but I'd say it reinforces my main point.

Good point, so the only real differentiator would be the size & quality of the data being fed and the fine tuning done on the model? I wonder what else differentiates LLMs from each other

Re: Training LLMs from ground zero as a startup

#50

Post by [deleted] »

twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

[deleted]
Post reply on HN