Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

271–280 of 311 posts

Re: Meta AI releases Code Llama 70B

#271

Post by blitzar »

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

Facebook went all in on the metaverse and turned into Meta; quite rightly, the market looked at what they produced for 10's of billions and decided their company was worthless.

Then Ai sprung to the front pages and any CEO who stood up and said "Ai" was rewarded with a 10x stock price. The unloved stepchild that was the ML team became the A team and the metaverse team have been sent to the naughty step. Facebook/Meta have no actual customer facing use for Ai unlike Microsoft/Google/GitHub but they like a good stonk price rise and so what we see is their stategy to stay in the ai game and relevant.

It turns out it is pretty good for the rest of us (possibly the first time facebook has given something positive to humanity) as we get shinny toys to play with.

Re: Meta AI releases Code Llama 70B

#273

Post by sa-code »

sestinj wrote: post #224
speedgoose wrote: post #179

Earlier quoted context omitted.

Continue doesn’t support tab completion like Copilot yet. A pull/merge request is being worked on: https://github.com/continuedev/continue/pull/758

Release coming later this week!

Thank you for your efforts

Re: Meta AI releases Code Llama 70B

#274

Post by MacsHeadroom »

pandominium wrote: post #228

Everyone is mentioning using 4090 and a smaller model, but I rarely see an analysis where the energy consumption is used. I think Copilot is already highly subsidized by Microsoft. Let's say you use Copilot around 30% of your daily work hours. How much kWh does an opensource 7B or 13B model use then in a month on one 4090? EDIT: I think for a 13B at 30% use per day it comes around 30$/no on energy bill. So probably w…

Using a model 30% of the day is only maybe 100 instances of use, with each lasting for about 6 seconds.

So really you're looking at using the GPU for around 10 minutes a day.

Monthly cost is pennies.

Re: Meta AI releases Code Llama 70B

#275

Post by d_sc »

sestinj wrote: post #229
dan_can_code wrote: post #226

Earlier quoted context omitted.

How do you test a 70B model locally? I've tried to query, but the response is super slow.

Personally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, you can get down to 2), but it would still require a higher-end Mac. Highly recommend Together, it runs quite quickly and is $0.9/million tokens

are there any docs on setting up togetherAI with continue.dev? would be interested in checking that out as an alternative to OpenAI for experimenting with larger models that won't run/run well on a m1 max.

Re: Meta AI releases Code Llama 70B

#278

Post by xigoi »

colesantiago wrote: post #132
Philpax wrote: post #91

Earlier quoted context omitted.

That may be true, but it's largely irrelevant. The ML framework in use has no bearing on whether or not you have the data required to reproduce the model being trained with that framework.

Do you and the GP have 350K GPUs and quality data to reproduce 1:1 whatever Facebook releases in their repos? Even if you want to reproduce the model and they give you the data, you would need to do this at Facebook scale, so you and the GP are just making moot points all around. https://about.fb.com/news/2023/05/metas-infrastructure-for-a... https://www.theregister.com/2024/01/20/metas_ai_plans/ The fact that these…

If you don’t have a machine powerful enough to compile some heavy AAA game, does that make the game open source?

Re: Meta AI releases Code Llama 70B

#279

Post by caeril »

dan_can_code wrote: post #226
sestinj wrote: post #171

Earlier quoted context omitted.

I’ve been working on continue.dev, which is completely free to use with your own Ollama instance / TogetherAI key, or for a while with ours. Was testing with Codellama-70b this morning and it’s clearly a step up from other OS models

How do you test a 70B model locally? I've tried to query, but the response is super slow.

Yeah, CPU inference is incredibly slow, especially as the context grows. 4-bit quantized on an A6000 should in theory work.

If those rent-seeking bastards at NVidia hadn't killed NVL on the 4090, you could do it on two linked 4090s for only $4k, but we have to live under the thumb of monopolists until such time as AMD 1. catches up on hardware and 2. fixes their software support.

Re: Meta AI releases Code Llama 70B

#280

Post by frognumber »

nullc wrote: post #255
frognumber wrote: post #253

Earlier quoted context omitted.

For reference for readers. SUPPORTED ========= * Ada / Hopper / A4xxx (but not A4000) * Ampere / A3xxx * Turing / Quadro RTX / GTX 16xx / RTX 20XX / Volta / Tesla EOL 2023/2024 ============= * Pascal / Quadro P / Geforce GTX 10XX / Tesla Unsupported =========== * Maxwell * Kepler * Fermi * Tesla (yes, this one pops up over and over, chaotically) * Curie Older don't really do GPGPU much. The older cards are also quite…

P40 is essentially a faster 1080 with 24GB ram. For many tasks (including LLMs) it's easy to be memory bandwidth bottlenecked and if you are they are more evenly matched. (newer hardware has more bandwidth, sure but not in a cost proportional manner). I find that my hosts using 9x P40 do inference on 70b models MUCH MUCH faster than a e.g. a dual 7763 and cost a lot less. ... and can also support 200B parameter model…

Interesting.

Technically, P40 is rated at an impressive 347.1GB/sec memory bandwidth, and 4060, at a slightly lower 272GB/sec. For bandwidth-limited workloads, the P40 still wins.

The 4090 is about 3-4x that, but as you point out, is not cost-competitive.

What do you use to fit 9x P40 cards in one machine, supply them with 2-3kW of power, and keep them cooled? Best I've found are older rackmount servers, and the ones I was looking at stoped short of that.

Post reply on HN