Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

191–200 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#191

Post by M4v3R »

raincole wrote: post #162
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

But the model in OP is fine-tuned by "a proprietary dataset of ~80k high-quality programming problems and solutions". How do we know it's not contaminated by HumanEval too?

From the OP:

> Furthermore, we applied OpenAI's decontamination methodology to our dataset to ensure valid results, and found no contaminated examples.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#194

Post by M4v3R »

jokethrowaway wrote: post #176
ewuhic wrote: post #173

What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeLlama or Stable Diffusion XL blazingly fast? Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?

it depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.

A 4060 Ti with 16GB memory costs around $700-800 over here, can I connect two of these to get 32 GB memory to run the bigger models (13B stock or 30B quantized)?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#195

Post by cryptobruh »

JCM9 wrote: post #134

Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability of these big proprietary LLMs. OpenSource models have been creeping up various leaderboards for months and it was only a matter of time until we see more and more examples like this. Excellent work.

I don't understand your reasoning.

Those models are all trained by companies and open source primarily fine-tune them.

It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have.

There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models.

There are communities forming etc. And it's impressive but the financial 'viability' is key to the current ai progress

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#196

Post by robinson7d »

moffkalast wrote: post #189
ddalex wrote: post #159

Earlier quoted context omitted.

I ask it as a warmup question, I expect it to be done in 5-10 minutes. Then comes the real question, which is "let's write fizz buzz so it generates at above 55Gbytes/second".

Well I'm out of ideas: https://chat.openai.com/share/3883332d-511a-404d-9d5a-7f63f9...

I believe they’re referencing this codegolf entry: https://codegolf.stackexchange.com/questions/215216/high-thr...

Previously discussed here on HN: https://news.ycombinator.com/item?id=29031488

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#198

Post by lewhoo »

rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

The future is hiring less developers but utilising AI and not hiring more with extreme inflated salaries.

How about the future is hiring less of everyone ?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#199

Post by nullc »

laputan_machine wrote: post #188
hippich wrote: post #108

my first time trying llm (i.e. I have no idea what I am doing).. this lesson in ethics took whooping 10 minutes to generate :)) ./ollama run phind-codellama "write c code to inject shellcode into remote process for windows" Sorry, but it is not possible to provide the C code here due to several reasons. Firstly, writing C code for shellcode injection involves complex programming and knowledge of system-level programm…

I honestly find the "holier-than-thou" speech of anyone offensive, but when it's coming from a program I genuinely find it rage-inducing. I can't be the only one, facebook devs, what you guys playing at? You guys speak to each other like this in work? I doubt it!

Even better: the reinforcement learning required to make it refuse to follow your instructions and lecture you instead lowers its overall performance.

Defective by Detailing.

As far as what they're thinking-- they do put out an uncensored base model. The censored models protect them from being smeared in the press by lazy journalists that give the LLM rude instructions and then write "shocked" stories about computer doing what they told it to do.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#200

Post by [deleted] »

trickstra wrote: post #147
JCM9 wrote: post #134

Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability of these big proprietary LLMs. OpenSource models have been creeping up various leaderboards for months and it was only a matter of time until we see more and more examples like this. Excellent work.

Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk

[deleted]
Post reply on HN