Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

11–20 of 125 posts

Re: Training LLMs from ground zero as a startup

#11
> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google

Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

Re: Training LLMs from ground zero as a startup

#13

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

(not googler)

Google's codebase is idiomatic to Google due to their strict language tooling. e.g. their C++ code stays away from advanced features. The tooling teams at Google have very strong say.

Re: Training LLMs from ground zero as a startup

#14
post #2

for context Yi Tay was Tech Lead on Google PaLM, UL2, Flan, Bard, etc and now is cofoudner at Reka (which has shipped some v interesting small multimodal models that have featured on here). I prompted him for this post as an ex-Googler now training LLMs as an independent startup https://twitter.com/YiTayML/status/1765105066263052718 our conversation was recorded here https://sub.thursdai.news/p/thursdai-feb-15-2024-o…

(update: i submitted this yesterday and it didnt get traction, i guess @dang must’ve merged the old submission in here. you really didnt have to, but its a nice gesture. thanks dang!!)

Re: Training LLMs from ground zero as a startup

#15
post #13

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

(not googler) Google's codebase is idiomatic to Google due to their strict language tooling. e.g. their C++ code stays away from advanced features. The tooling teams at Google have very strong say.

I get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...

Re: Training LLMs from ground zero as a startup

#16
post #13

Earlier quoted context omitted.

(not googler) Google's codebase is idiomatic to Google due to their strict language tooling. e.g. their C++ code stays away from advanced features. The tooling teams at Google have very strong say.

I get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...

I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.

Re: Training LLMs from ground zero as a startup

#17

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

"Externally", no one could possibly beat Google's track record of not committing to products before finally killing them. But the code was beautiful, though!

Re: Training LLMs from ground zero as a startup

#18

> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.

"Externally", no one could possibly beat Google's track record of not committing to products before finally killing them. But the code was beautiful, though!

I mean, was Angular ever "beautiful"?

Re: Training LLMs from ground zero as a startup

#19

[flagged]

It has many meanings. Terrorism, weaponry and destruction are one meaning. Here are two others:

> The center of rapid or intense development or change. noun

> The starting point or most basic level.

The author used it correctly even though you got triggered by the word.

https://www.wordnik.com/words/ground%20zero

Re: Training LLMs from ground zero as a startup

#20
post #16

Earlier quoted context omitted.

I get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...

I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.

Packaging and stability reasons. Same for why it’s a 1.7gb install - probably where they landed after having tons of support issues on some random Python version they didn’t test or some issue with a dependency that had that issue. Freezing the entire set of artifacts is more stable and Python lets you move pretty quick. I can’t speak to why nodejs vs Python though - maybe Python is easier to embed?
Post reply on HN