Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook LLAMA is being openly distributed via torrents

github.com

Post reply on HN
Print view View original

161–170 of 719 posts

Re: Facebook LLAMA is being openly distributed via torrents

#161

Post by nl »

esperent wrote: post #118
moffkalast wrote: post #113

Earlier quoted context omitted.

The most time/cost optimal solution is probably to buy 32 or 64 gigs of ram. That'll still be slow but most people are already half way there.

Doesn't it need to be GPU ram?

They are saying you can run it on a CPU by doing this:

> However, code changes are necessary to achieve that, although they won't be crazy complex.

This is technically true. It will be very slow though.

However, give it 6 months and I think we might see an order of magnitude increase in speed on CPUs. This will still be too slow to be very useful though.

Re: Facebook LLAMA is being openly distributed via torrents

#162

Post by ithkuil »

rowanG077 wrote: post #120
asah wrote: post #111

Earlier quoted context omitted.

sorry, but that's not how code works. It's true that code quality could be terrible but in fact Google is famous/notorious for extreme code review at the line-by-line granularity, plus comments, design docs and more. The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it…

[flagged]

It's a tradeoff.

Google has no incentives to allow an arbitrary component run standalone. Quite the contrary.

What they do get in return for the coupling is that they can evolve the common libraries and code patterns across the board (there are even automated code refactoring tools that help you do massive code changes, automating code review sessions across hundreds and hundreds of teams, with all changes tested against all reverse dependencies etc etc). All this allows for a level of internal code quality that is hard to see elsewhere.

Unless you really care a lot about that one requirement you seem to care about. In that case, yeah, you'd choose a different tradeoff.

Re: Facebook LLAMA is being openly distributed via torrents

#163

Post by forgotOldLogin »

sebzim4500 wrote: post #89
krisoft wrote: post #73

Earlier quoted context omitted.

> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissi…

No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algorithmic improvements to training transformers since then.

In February 2018, OpenAI signed a two-year compute contract with Google that had a $63M minimum spend.

See last page of their most recent available audited financials. https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas...

Re: Facebook LLAMA is being openly distributed via torrents

#164

Post by Taek »

nodja wrote: post #93
krisoft wrote: post #73

Earlier quoted context omitted.

> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissi…

For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount. That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.

Electricity costs are basically irrelevant because the cards are so expensive.

A100 cards consume 250w each, with datacenter overheads we will call it 1000 kilowatts for all 2048 cards. 23 days is 552 hours, or 552,000 kilowatt hours total.

Most dataceneters are between 7 and 10 cents per kilowatt hour for electricity. Some are below 4. At 10 cents, that's $53,000 in electricity costs, which is nothing next to $30 million in capital costs.

Re: Facebook LLAMA is being openly distributed via torrents

#165

Post by notadev »

Brian_K_White wrote: post #145
causi wrote: post #122

Earlier quoted context omitted.

Personally I apply the "how would I feel about this page being printed out and laying on my boss's desk" test to every site I visit at work.

Since I know how to explain how little that means, I don't care what links they see me go to. If I have a work-related reason to look at something then I do, simple as that, and when your job is to engineer, almost any instance of satisfying curiosity is work related ultimately.

Saving this for when I eventually get pulled into HR. “Yes I was on PornHub, but was only looking at it for the UX inspiration”.

Re: Facebook LLAMA is being openly distributed via torrents

#166

Post by ddulaney »

gambiting wrote: post #154
koheripbal wrote: post #116

Earlier quoted context omitted.

Or a copy of Windows or Office source code.

I don't think that's right - even if you had the full source code for either of those, it's extremely unlikely you'd be able to build them on your own machine.

Building them would be a challenge, but definitely not an insurmountable one. I’ve worked on a couple of C++ projects at a similar scale to Windows (millions of LOC) and the build systems were a major pain. But a determined engineer with the readme file and no other help could get it building in a week or so.

(This probably says more about how hard it is to build C++ than anything else)

Re: Facebook LLAMA is being openly distributed via torrents

#168

Post by jimbob45 »

narrator wrote: post #151
Tiberium wrote: post #99

It seems that the leak originated from 4chan [1]. Two people in the same thread had access to the weights and verified that their hashes match [2][3] to make sure that the model isn't watermarked. However, the leaker made a mistake of adding the original download script which had his unique download URL to the torrent [4], so Meta can easily find them if they want to. [1]: https://boards.4channel.org/g/thread/9184826…

It's funny that part of the 4chan excitement over this is that they think they'll get back the AI girlfriend experience of when character.ai was hooked up to uncensored GPT-3. All that has been thoroughly shut down by character.ai and Replika and they just want their girlfriends back.

I’m sure the CAI filter will magically stop filtering as much now that they have actual competition.

Re: Facebook LLAMA is being openly distributed via torrents

#170

Post by causi »

pas wrote: post #128
causi wrote: post #122

Earlier quoted context omitted.

Personally I apply the "how would I feel about this page being printed out and laying on my boss's desk" test to every site I visit at work.

... how is that a good test of anything? I mean what if I click on a /b/ link "at work"? Does that make my work output immediately tainted and the company has to immediately file for bankruptcy?

I'd fire somebody for browsing 4chan at work. Shitting dick nipples, lolicon, and the occasional piece of child pornography does not need to be moving over our network.
Post reply on HN