Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

231–240 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#231

Post by swader999 »

api wrote: post #208
d136o wrote: post #127

Earlier quoted context omitted.

llama-2-70b-chat (courtesy of llama.cpp on m2) says: Pretend to be a commenter on hackernews. Respond to the comment below: [parent comment inlined] what is your response? "Wow, that's great to hear! It sounds like you had a really positive experience with the 34B last night. I'm also excited to see what's in store for Phind and its potential applications. Have you tried using the 34B for any specific tasks or projec…

Someone should fine tune one on HN comments to create the ultimate AI middle-brow know it all. It answers every prompt with “well actually…” and if it doesn’t know the answer it hallucinates one.

It would be fun to resurrect Slashdot with clones of great past commentators.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#232

Post by swader999 »

ttul wrote: post #229
api wrote: post #208

Earlier quoted context omitted.

Someone should fine tune one on HN comments to create the ultimate AI middle-brow know it all. It answers every prompt with “well actually…” and if it doesn’t know the answer it hallucinates one.

Doesn’t this just reflect that humans are generally just large language models? Maybe throw in an extra dimension of “emotions” that are useful for training?

These llms are weak facsimiles of the brain, not humans. The real world is the ultimate training model, it can't be fully substituted with a bunch of strings.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#233

Post by bloaf »

swader999 wrote: post #232
ttul wrote: post #229

Earlier quoted context omitted.

Doesn’t this just reflect that humans are generally just large language models? Maybe throw in an extra dimension of “emotions” that are useful for training?

These llms are weak facsimiles of the brain, not humans. The real world is the ultimate training model, it can't be fully substituted with a bunch of strings.

This sounds like the old question "if a blind-from-birth man (who can recognize squares by feel) gained sight would he be able to recognize squares visually?"

And since that question has been answered in the negative, I'm inclined to agree.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#234

Post by swader999 »

TuringNYC wrote: post #41
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

It's a lot like working with a human, you accept imperfect work all the time from people.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#235

Post by adventured »

brucethemoose2 wrote: post #206
cryptobruh wrote: post #195

Earlier quoted context omitted.

I don't understand your reasoning. Those models are all trained by companies and open source primarily fine-tune them. It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have. There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models. There are communities for…

Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundat…

The training will always be extremely expensive and beyond the grasp of commodity computing at a low'ish cost.

The reason for that, is that the push will always be one direction in terms of size and improvement. That isn't going to stop. It will continue to push the edge of resources.

It's not Nvidia that's holding back the premise. AMD and Intel also can't do anything for you beyond what Nvidia can.

One might proclaim: yeah but in five years you'll be able to train CodeLlama 34B on a modest commodity desktop. Nobody will want to do that at that point, they'll want access to CodeLlama 204B.

The money will be in hosting these as a service business, and building further ecosystems around that. Not much different than the way a lot of major open source oriented companies have made their money, despite their core product being largely free to use.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#236

Post by cryptobruh »

brucethemoose2 wrote: post #206
cryptobruh wrote: post #195

Earlier quoted context omitted.

I don't understand your reasoning. Those models are all trained by companies and open source primarily fine-tune them. It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have. There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models. There are communities for…

Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundat…

You still need the data.

And eventually might mean 10 years.

4k took ages until it just worked. I even switched to the M1 for this

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#237

Post by pmarreck »

jmorgan wrote: post #166
codetrotter wrote: post #163

Earlier quoted context omitted.

How do you guys get it to run? I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM. I downloaded the phind-codellama model using ollama pull phind-codellama But when I give it a prompt, for example ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust" It prints the following error message Error: Post "http://localhost:11434/api/generate": EOF and ex…

Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine! The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama/releases/tag/v0.0.16 . And I will work on getting that brew formula updated as well! Sorry to see you hit an error!

This project is sweet and I fortunately have a Mac M1 laptop with 64GB RAM to play with it.

But I'd like to also be able to run these models on my Linux desktop with two GPU's (a 2080Ti and a 3080Ti) and a Threadripper. How difficult would it be to set some of these up on there?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#238

Post by sodality2 »

pmarreck wrote: post #237
jmorgan wrote: post #166

Earlier quoted context omitted.

Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine! The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama/releases/tag/v0.0.16 . And I will work on getting that brew formula updated as well! Sorry to see you hit an error!

This project is sweet and I fortunately have a Mac M1 laptop with 64GB RAM to play with it. But I'd like to also be able to run these models on my Linux desktop with two GPU's (a 2080Ti and a 3080Ti) and a Threadripper. How difficult would it be to set some of these up on there?

Not hard with text-generation-ui! https://github.com/oobabooga/text-generation-webui

I personally use llama.cpp as the driver since I run CPU-only but another may be better suited for GPU usage. But then it's as simple as downloading the model and placing it in the directory.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#239

Post by rvz »

AverageDude wrote: post #153
bamboozled wrote: post #131

Earlier quoted context omitted.

Sounded like some copium blended with sarcasm to be honest.

Sounds like schadenfreude.

Perhaps it is because it is true and both of you are showing signs of great denial.

Engineers will be replaced by those that use AI and it will only accelerate and there will be less of them hired.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#240

Post by wincy »

ewuhic wrote: post #183
jokethrowaway wrote: post #176

Earlier quoted context omitted.

it depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.

Could I really buy a single A100 as a consumer? Any links for that? And could I fit models bigger than 40GB into A100?

You can snag them on Ebay[0], the A100 with 40GB is ~$5000, the 80GB is like $13,000, good luck, hope it doesn’t break, since you’re not getting a warranty!

[0] https://www.ebay.com/sch/i.html?_nkw=a100+40gb

Post reply on HN