21–30 of 125 posts
[flagged]
I checked a few dictionaries and the second or third sense for all was "the very beginning or most elementary level" or "starting from scratch".
I do agree Ground Zero is a loaded term, but it has an existing meaning and it is the above one I think of first when hearing the term.
Its like the word holocaust. We use it in phases like "a nuclear holocaust" while at the same time it refers to a specific horrible incident.
Earlier quoted context omitted.
I get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...
I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.
Training LLM from scratch is a super important issue that affects the pace and breadth of iteration of AI almost as much as the raw hardware improvements do. The blog is fun but somewhat shallow and not technical or very surprising if you’ve worked with clusters of GPUs in any capacity over the years. (I liked the perspective of a former googler, but I’m not sure why past colleagues would recommend Jax over pytorch f…
Earlier quoted context omitted.
I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.
Packaging and stability reasons. Same for why it’s a 1.7gb install - probably where they landed after having tons of support issues on some random Python version they didn’t test or some issue with a dependency that had that issue. Freezing the entire set of artifacts is more stable and Python lets you move pretty quick. I can’t speak to why nodejs vs Python though - maybe Python is easier to embed?
[flagged]
Perhaps dramatically ironic, but this is precisely the kind of thing that a LLM would help with. The opening paragraph uses "from scratch" which is a more appropriate choice also.
for context Yi Tay was Tech Lead on Google PaLM, UL2, Flan, Bard, etc and now is cofoudner at Reka (which has shipped some v interesting small multimodal models that have featured on here). I prompted him for this post as an ex-Googler now training LLMs as an independent startup https://twitter.com/YiTayML/status/1765105066263052718 our conversation was recorded here https://sub.thursdai.news/p/thursdai-feb-15-2024-o…
(update: i submitted this yesterday and it didnt get traction, i guess @dang must’ve merged the old submission in here. you really didnt have to, but its a nice gesture. thanks dang!!)
So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these natur…
I don't think most of them have any kind of secret sauce. I think the founders hope to get bought out simply for being able to train "near-SOTA" LLMs. I guess achieving that level of skill and infra could be valuable enough to build upon.
Earlier quoted context omitted.
"Externally", no one could possibly beat Google's track record of not committing to products before finally killing them. But the code was beautiful, though!
I mean, was Angular ever "beautiful"?
Earlier quoted context omitted.
I get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...
I have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.
For anything more advanced they offer language specific SDKs in Rust, Swift, Kolton, etc…
For example integrating storage in an iOS app.