141–150 of 719 posts
Earlier quoted context omitted.
Fortune favors the brave
Personally I apply the "how would I feel about this page being printed out and laying on my boss's desk" test to every site I visit at work.
Earlier quoted context omitted.
sorry, but that's not how code works. It's true that code quality could be terrible but in fact Google is famous/notorious for extreme code review at the line-by-line granularity, plus comments, design docs and more. The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it…
[flagged]
The infrastructure in these companies is a huge amount of scaffolding that's non-trivial to replicate.
Earlier quoted context omitted.
For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount. That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.
$30m training cost is too high. Amazon's p4d.24xlarge is $32.77 an hour for 8 A100 GPUs. 2048 A100 GPUs for 23 days costs $4.6m at that rate. You might even get a discount.
Earlier quoted context omitted.
It's not even clear someone has leaked the models. A random person has put a download link on a PR, it could be anything.
The folder structure definitely looks like model weights, I didn't download or run it though so for all I know it only generates the words to "Never Gonna Give You Up".
Earlier quoted context omitted.
Fortune favors the brave
Personally I apply the "how would I feel about this page being printed out and laying on my boss's desk" test to every site I visit at work.
Earlier quoted context omitted.
Personally I apply the "how would I feel about this page being printed out and laying on my boss's desk" test to every site I visit at work.
... how is that a good test of anything? I mean what if I click on a /b/ link "at work"? Does that make my work output immediately tainted and the company has to immediately file for bankruptcy?
It’s a little like the Van Halen M&M test: if you don’t follow that rule, people have to wonder what other expectations you won’t meet.
Earlier quoted context omitted.
Personally I apply the "how would I feel about this page being printed out and laying on my boss's desk" test to every site I visit at work.
... how is that a good test of anything? I mean what if I click on a /b/ link "at work"? Does that make my work output immediately tainted and the company has to immediately file for bankruptcy?
No, much easier just to fire you. You’ve shown a disregard for what is almost certainly company policy, and created legal risks for the company (e.g. if anyone walked in on you while your screen was showing naked people).
Earlier quoted context omitted.
> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissi…
No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algorithmic improvements to training transformers since then.
You are right! Wow. Thank you for correcting me.
> GPT-3 probably cost under $5 million,
Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?