Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

241–250 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#241

Post by wincy »

M4v3R wrote: post #194
jokethrowaway wrote: post #176

Earlier quoted context omitted.

it depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.

A 4060 Ti with 16GB memory costs around $700-800 over here, can I connect two of these to get 32 GB memory to run the bigger models (13B stock or 30B quantized)?

A 3090 with 24GB of vram is $650 used on Facebook where I live. I think that would be a better card as you can use nvlink.

Alternatively, the 4000 series cards rely on the PCI express lanes for linking, so if you want to get the benefits of linking the 4000 series cards as a consumer you’ll want a Threadripper or even maybe an EPYC CPU/motherboard combo.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#242

Post by dontupvoteme »

wordpad25 wrote: post #143
dontupvoteme wrote: post #91

Earlier quoted context omitted.

You can do that as well, under your complete control. That's a framework they put around the model.

"model" is end-to-end, input-to-output, inclusive of the entire framework and it's guardrails and everything else if they are able to detect hallucinations, filter them out and automatically re-run, that's a huge improvement in result, even though core model didn't get new training

That's the product. The model is the kernel of the product.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#243

Post by tomduncalf »

JoelJacobson wrote: post #222
jquery wrote: post #219

Earlier quoted context omitted.

>3 parenthesis quote Do you mean triple double quotes?

That's what I've been using but still get that error.

Same here, would be interested to know if there's a solution!

I created an issue, if you have an example prompt to add that would be helpful! https://github.com/jmorganca/ollama/issues/422

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#244

Post by progbits »

cryptobruh wrote: post #236
brucethemoose2 wrote: post #206

Earlier quoted context omitted.

Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundat…

You still need the data. And eventually might mean 10 years. 4k took ages until it just worked. I even switched to the M1 for this

4k as in screen resolution? That was a thing for many many years before M1.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#245

Post by coder543 »

M4v3R wrote: post #194
jokethrowaway wrote: post #176

Earlier quoted context omitted.

it depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.

A 4060 Ti with 16GB memory costs around $700-800 over here, can I connect two of these to get 32 GB memory to run the bigger models (13B stock or 30B quantized)?

One additional caveat worth considering... a lot of LLM computation is often memory bandwidth bound. The 4060 Ti is infamous for how deeply Nvidia cut the memory system. The 3060 Ti had a 256-bit bus that was capable of 448GB/s of memory bandwidth. The 4060 Ti has a paltry 128-bit bus that can only do 288GB/s, about 64% of the memory bandwidth of the previous generation.

For comparison, an RTX 3090 has 935GB/s of memory bandwidth. As the other person mentioned, that would likely be a much better card if you can find a used one for a reasonable price... but that's just my opinion.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#246

Post by slashdev »

bloaf wrote: post #233
swader999 wrote: post #232

Earlier quoted context omitted.

These llms are weak facsimiles of the brain, not humans. The real world is the ultimate training model, it can't be fully substituted with a bunch of strings.

This sounds like the old question "if a blind-from-birth man (who can recognize squares by feel) gained sight would he be able to recognize squares visually?" And since that question has been answered in the negative, I'm inclined to agree.

Has it been answered in the negative?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#247

Post by timthelion »

wincy wrote: post #240
ewuhic wrote: post #183

Earlier quoted context omitted.

Could I really buy a single A100 as a consumer? Any links for that? And could I fit models bigger than 40GB into A100?

You can snag them on Ebay[0], the A100 with 40GB is ~$5000, the 80GB is like $13,000, good luck, hope it doesn’t break, since you’re not getting a warranty! [0] https://www.ebay.com/sch/i.html?_nkw=a100+40gb

I'm seeing a bunch of results from China. I was under the impression it was illegal to sell A100 to China, so where are the chinese sellers getting these cards?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#248

Post by lewhoo »

rvz wrote: post #239
AverageDude wrote: post #153

Earlier quoted context omitted.

Sounds like schadenfreude.

Perhaps it is because it is true and both of you are showing signs of great denial. Engineers will be replaced by those that use AI and it will only accelerate and there will be less of them hired.

Engineers will be replaced by those that use AI

By that time not only engineers will get replaced. What do you do if I may ask ?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#249

Post by ResearchCode »

EvgeniyZh wrote: post #155
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

I have a several arguments why contamination is probably not the main reason of performance difference. When we worked on StarCoder, people ran gpt-4 on MultiPL-E, which doesn't have canonical solutions in the internet, and the performance was higher that what you would expect from official numbers Official contamination analysis shows only minor drop in performance even though contamination is fairly high (you may a…

The "intelligence" of large language models needs to be evaluated like the abilities of self-proclaimed psychics. You send your binary to an independent third party and who evaluates it on new problems. It's only a "Human eval" once.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#250

Post by plandis »

syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

What kinds of questions do you ask these?
Post reply on HN