Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

Post reply on HN
Print view View original

31–40 of 125 posts

Re: Training LLMs from ground zero as a startup

#31

Post by twelfthnight »

anonymous-panda wrote: post #20
jen20 wrote: post #16

Earlier quoted context omitted.

I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.

Packaging and stability reasons. Same for why it’s a 1.7gb install - probably where they landed after having tons of support issues on some random Python version they didn’t test or some issue with a dependency that had that issue. Freezing the entire set of artifacts is more stable and Python lets you move pretty quick. I can’t speak to why nodejs vs Python though - maybe Python is easier to embed?

Yeah, I imagine that was the decision calculus. "Instead of spending some more effort to save millions of unnecessary downloads of python's runtime using a different language, let's just bundle Python!"

I wouldn't be surprised if it was version 2.7 too...

Re: Training LLMs from ground zero as a startup

#32

Post by observationist »

anonymous-panda wrote: post #19
observationist wrote: post #12

[flagged]

It has many meanings. Terrorism, weaponry and destruction are one meaning. Here are two others: > The center of rapid or intense development or change. noun > The starting point or most basic level. The author used it correctly even though you got triggered by the word. https://www.wordnik.com/words/ground%20zero

Not necessarily. In the context you reference, you could say Silicon Valley is ground zero for technological revolutions, or the explosive spread of social media. Deliberately building a thing from the ground up doesn't match the nuances of the many and varied meanings of the phrase; it's not quite right.

The primary semantic connection is the misuse of the phrase "from the ground up" and conflating it with "ground zero." There's no direct or indirect semantic relationship between something explosive or chaotic or destructive, and deliberately and methodically building a language model, in a calm and rational process. The meaning doesn't map to the phrase in the context of this article's title.

If you were building an etymology website from the ground up, you'd hope it would be ground zero for an explosion of awareness about how different words, phrases, and sayings could be used correctly in different contexts.

Something like that - I was trying to be considerate about the Ground Zero memorial, and the many ways in which "ground zero" can refer to things that people are deeply invested in. My personal stake is just an observation that it's not quite the right turn of phrase for the use to which it was put.

It seems likely that "from the ground up" was the intended phrase, and it might have gotten awkwardly revised, since the word startup is also in the title?

Re: Training LLMs from ground zero as a startup

#33

Post by nyokodo »

pests wrote: post #21
observationist wrote: post #12

[flagged]

I think you are incorrect? I checked a few dictionaries and the second or third sense for all was "the very beginning or most elementary level" or "starting from scratch". I do agree Ground Zero is a loaded term, but it has an existing meaning and it is the above one I think of first when hearing the term. Its like the word holocaust. We use it in phases like "a nuclear holocaust" while at the same time it refers to…

> We use it in phases like "a nuclear holocaust" while at the same time it refers to a specific horrible incident.

The historical and still sometimes contemporary meaning of "holocaust" is a burnt offering at an altar. But, if you're using that term now in a general context to mean merely a burnt offering it is so overcome by historical events that you will confuse your audience, cloud out your message, and probably cause widespread offense. Well if there were similar more recent examples of that in english it would be key vocabulary surrounding September 11 2001, like "Ground Zero." Therefore, while the OP is technically incorrect about the dictionary definition of the phrase, they aren't wrong about the meaning of the phrase to contemporary english speakers.

Re: Training LLMs from ground zero as a startup

#34

Post by titanomachy »

twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

I thought the quality was pretty high, largely because there were a lot of rails constraining how code should be written. Most of the code I dealt with was written using somewhat rigid (but generally well-designed) frameworks with programmatically-enforced style guides.

Also, most work seemed to involve some balance of junior and more experienced people, which helped keep quality higher. Outside of Google, I've seen pretty large projects written by new grads with little supervision (and on a tight timeline). Those codebases can be pretty hairy.

Re: Training LLMs from ground zero as a startup

#35

Post by pests »

nyokodo wrote: post #33
pests wrote: post #21

Earlier quoted context omitted.

I think you are incorrect? I checked a few dictionaries and the second or third sense for all was "the very beginning or most elementary level" or "starting from scratch". I do agree Ground Zero is a loaded term, but it has an existing meaning and it is the above one I think of first when hearing the term. Its like the word holocaust. We use it in phases like "a nuclear holocaust" while at the same time it refers to…

> We use it in phases like "a nuclear holocaust" while at the same time it refers to a specific horrible incident. The historical and still sometimes contemporary meaning of "holocaust" is a burnt offering at an altar. But, if you're using that term now in a general context to mean merely a burnt offering it is so overcome by historical events that you will confuse your audience, cloud out your message, and probably…

> it is so overcome by historical events that you will confuse your audience

I really don't think so. I think almost every US president has used that exact phrase ("nuclear holocaust") in the last 20 years. Trump, Biden, Obama, Bush, have all casually used the phase "nuclear holocaust" while speaking to the American public.

> they aren't wrong about the meaning of the phrase to contemporary english speakers.

I am a contemporary English speaker, I grew up during 9/11. "Ground Zero" as a noun means a reference to 9/11. Using it normally like "starting from ground zero" is completely understandable and non-controversial to me. My opinion of course.

Re: Training LLMs from ground zero as a startup

#37

Post by anonymous-panda »

pests wrote: post #24
anonymous-panda wrote: post #20

Earlier quoted context omitted.

Packaging and stability reasons. Same for why it’s a 1.7gb install - probably where they landed after having tons of support issues on some random Python version they didn’t test or some issue with a dependency that had that issue. Freezing the entire set of artifacts is more stable and Python lets you move pretty quick. I can’t speak to why nodejs vs Python though - maybe Python is easier to embed?

What? They only get package and stability because they include the runtime. If they just went with a compiled language they could distribute native binaries and have actual packaging and stability.

Yes, but it’s not just a single metric. Another is how easy it is for them to hire productive members of the team and how much that costs them - middling Python developers churning out fine”ish” code are cheaper than Rust developers doing the same. It’s hard to find a language where you can be as productive as a developer in Python that also has AOT compilation to generate standalone binaries.

Tldr: there’s multiple factors to consider here and it’s more interesting to understand the pressures that cause the decisions, especially if you want to try to create a world where different decisions are made.

Re: Training LLMs from ground zero as a startup

#38

Post by marcyb5st »

twelfthnight wrote: post #15
winwang wrote: post #13

Earlier quoted context omitted.

(not googler) Google's codebase is idiomatic to Google due to their strict language tooling. e.g. their C++ code stays away from advanced features. The tooling teams at Google have very strong say.

I get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...

Did you install all the components? Because if so you also installed emulators for the pubsub and big table (maybe others, I don't remember) which explain the big footprint.

Re: Training LLMs from ground zero as a startup

#39

Post by dheera »

winwang wrote: post #13
twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

(not googler) Google's codebase is idiomatic to Google due to their strict language tooling. e.g. their C++ code stays away from advanced features. The tooling teams at Google have very strong say.

> e.g. their C++ code stays away from advanced features

Which honestly is a GOOD thing because it would make it much easier for newcomers to ramp up on existing codebases. Most people aren't used to working with spaceships and constexprs.

Readability is also far more valuable to a large team than efficiency for anything that isn't a number-crunching loop.

Re: Training LLMs from ground zero as a startup

#40

Post by twelfthnight »

titanomachy wrote: post #34
twelfthnight wrote: post #11

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

I thought the quality was pretty high, largely because there were a lot of rails constraining how code should be written. Most of the code I dealt with was written using somewhat rigid (but generally well-designed) frameworks with programmatically-enforced style guides. Also, most work seemed to involve some balance of junior and more experienced people, which helped keep quality higher. Outside of Google, I've seen…

That honestly does seem like a recipe for good code. And sure, there's tons of open source out there of dubious quality.

@resource0x in a sibling comment made the point that it's possible to write great code even if the program is a flawed design. I'm probably conflating those things.

Post reply on HN