Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

171–180 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#171

Post by az226 »

TuringNYC wrote: post #41
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

Quality has degraded but token generation speed has increased. The GPT4 of today isn’t the same as it used to be. To get the old slow model you need to use GPT4-0314, that gives higher quality answers like it used to.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#172

Post by jdthedisciple »

dado3212 wrote: post #149
trickstra wrote: post #147

Earlier quoted context omitted.

Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk

Is this really the line we want to draw in the sand? It’s not open source because it can’t be trivially used by AWS and Google? It feels like the popularism is the point, not the complete lack of restrictions.

Well, you see you are absolutely correct.

But unfortunately none of the "True Open Source" crusaders I have seen so far can think beyond their tribalism-based schism.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#173

Post by ewuhic »

What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeLlama or Stable Diffusion XL blazingly fast?

Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#174

Post by knotty66 »

Great stuff - and training was so cheap too - it would have cost less than $200 on Runpod, and half that price on spot instances.

I guess it's time languages other than Python, especially niche ones, started collating their own language specific datasets.

Personally, I daydream about an Elixir specific LLM I could run locally, trained or fine tuned to respond in an idiomatic fashion, and plug into a tool like Cursor.so.

Are there any examples of the internal dataset used in the 80K instruction / answer pairs Phind used to tune this?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#175

Post by JoelJacobson »

jmorgan wrote: post #166
codetrotter wrote: post #163

Earlier quoted context omitted.

How do you guys get it to run? I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM. I downloaded the phind-codellama model using ollama pull phind-codellama But when I give it a prompt, for example ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust" It prints the following error message Error: Post "http://localhost:11434/api/generate": EOF and ex…

Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine! The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama/releases/tag/v0.0.16 . And I will work on getting that brew formula updated as well! Sorry to see you hit an error!

Is there a limit on the number of lines/tokens it can take as input? Can it be increased? The limit seems to be somewhere above 30 lines.

I'm on a MacBook Pro M1 Max 64GB.

    % ollama --version
    ollama version 0.0.16
    % ollama run phind-codellama "$(cat prompt.txt)"
    ⠧   Error: Post "http://localhost:11434/api/generate": EOF
    % wc prompt.txt
         335    1481   11457 prompt.txt

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#176

Post by jokethrowaway »

ewuhic wrote: post #173

What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeLlama or Stable Diffusion XL blazingly fast? Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?

it depends how much you're willing to spend.

You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$.

You can build a normal pc with a 4090, otherwise you're in datacenter territory.

You can bundle up more GPUs.

You can decrease the precision of your models to get them to run on cards with less RAM.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#177

Post by somenameforme »

rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

I think a smart company plans for what they see the future as holding, as opposed to optimizing for what maximizing gains in the short term. And text generation programs seem to be following near to the exact same path as image generation programs did after DALL-E.

So in the future we're probably not going to see any single centralized player, but rather a wide decentralization with many options. It seems in this future, providing a 'decentralization ready' to go package that people can use and deploy for all sorts of things is far smarter than blowing through billions of dollars (and goodwill) desperately trying to become a centralized behemoth when you have no moat, no network effect, and perhaps a temporary technical advantage at best.

And while OpenAI has blown through a bunch of good will going from Open-ForSociety-AI to Closed-ForProfit-AI, Facebook has simultaneously gained a bunch of good will. What this translates to is really difficult to even imagine, but it's certainly desirable. And really I think this intangible of good will is something many modern corporations don't give enough value to, and it's hurting them - catching more flies with honey than vinegar, and all that.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#179

Post by nwoli »

haxiel wrote: post #133
rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

A previous post here on HN [1] discussed the leak of the original LLaMA model and Stable Diffusion. The article points out that the open-source community's effort resulted in more innovation, and an ecosystem built up around these models leading to rapid adoption. That gives more control to the original authors of the model against their competitors. I think Meta has recognized this and wants to keep their lead going…

They’re also already giving out millions of dollars in man hours in releasing and maintaining open source software, as well as lots of other much less used ML models. So they have an existing open source strategy and it makes sense to double down on this one since it’s clearly more impactful than a lot of the other ones

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#180

Post by nwoli »

tyhjnntyny wrote: post #21

This stuff is accelerating it alarming pace. I know of three or four off the shelf solutions to self host and run models now. I have learned so much and continue to learn about how this stuff works.

I don’t think it’s alarming I think it’s cool. It’s just autocomplete :)
Post reply on HN