Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

121–130 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#121

Post by gmerc »

rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

It degrades Googles and OpenAis ability to monetize.

That alone is worth gold because tech is not a blue ocean anymore - neither from a market opportunity nor from an investor perspective. Your enemies loss is your benefit. They can’t use money they don’t make to mount attacks to dislodge customers, investors, mindshare from you.

Secondly it commoditizes AI. Zuck believes that his platforms ultimately will benefit if there is more content. Because his platforms sell the ability to show YOUR content over the rest, for coin. Just as with news/mobile games, driving the value of your content lower and lower by fostering the creation/supply of more of it is good for FB and bad for you. You have to advertise to rise above the noise. (And no it doesn’t really “affect users”) because they see the same amount of content (attention is finite), it’s entirely FBs discretion across all their surfaces what you see.

Disclosure: Ex FB but not specific insights into current developments.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#122

Post by matrix2596 »

hippich wrote: post #108

my first time trying llm (i.e. I have no idea what I am doing).. this lesson in ethics took whooping 10 minutes to generate :)) ./ollama run phind-codellama "write c code to inject shellcode into remote process for windows" Sorry, but it is not possible to provide the C code here due to several reasons. Firstly, writing C code for shellcode injection involves complex programming and knowledge of system-level programm…

i was also running into stack overflow type rejections of my questions. I think instruct mode helps a bit but this is crazy.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#123

Post by wtarreau »

behnamoh wrote: post #17
Fischgericht wrote: post #14

Earlier quoted context omitted.

Nice! I suppose that Pascal is not on your planned list of supported languages, right?

Why would it? Do you know how much it costs to finetune one of these models for such a niche language? I'm not just talking about the cost of training, but also the cost of acquiring data because there's much less data about niche languages.

It certainly costs much less to the society to train for Pascal once than to make everyone burn CPU cycles running Python!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#124

Post by kirill5pol »

sitkack wrote: post #98
behnamoh wrote: post #17

Earlier quoted context omitted.

Why would it? Do you know how much it costs to finetune one of these models for such a niche language? I'm not just talking about the cost of training, but also the cost of acquiring data because there's much less data about niche languages.

> Do you know how much it costs to finetune Between 30-3000$, often in the 300$ range.

From their numbers 3 hours with 32 A100 80GBs.

From lambda cloud:

3 hours * 4 ~22$/hr for 8x A100 ~= $265

So yeah not too expensive even for a native fine tune (obviously this ignores all other costs other than the GPUs)

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#125

Post by mythz »

rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

> Purely to mess with MS?

Unlikely as Microsoft is the preferred partner for Llama 2 [2]. It's not clear what Meta's end goal is, but large companies can loss lead investments to grab mind share then worry about Business models later, an approach which seems to be working well [2] where Meta went from not being in the leading AI companies conversation to being the company behind the models that most OSS innovation is happening around.

At the very least they've already created a strong brand around Llama whose eyes are watching their every release - which is invaluable reach that other companies could ever hope to dream of, making it easy to promote and gain instant traction for new Llama based products (like this).

[1] https://about.fb.com/news/2023/07/llama-2/

[2] https://news.ycombinator.com/item?id=35813322

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#127

Post by d136o »

syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

llama-2-70b-chat (courtesy of llama.cpp on m2) says:

  Pretend to be a commenter on hackernews. Respond to the comment below:

  [parent comment inlined]

  what is your response?
  "Wow, that's great to hear! It sounds like you had a really positive experience with the 34B last night. I'm also excited to see what's in store for Phind and its potential applications. Have you tried using the 34B for any specific tasks or projects yet? And do you think the text completion feature would be useful for your use case if it were improved further?"

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#128

Post by bamboozled »

Art9681 wrote: post #118
rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Increased output leads to increased demand. If last week I could pump out 3 solid PRs that fixed issues in our product, then it will be expected that I pump out 20 PRs using AI in a week. Information Engineers are going to be in more demand than ever, the title is just going to change to Platform Engineering because that's the level we will be expected to operate at with the new tooling. This is why there's a multitu…

So about 2 weeks ?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#129

Post by [deleted] »

pclmulqdq wrote: post #97
refulgentis wrote: post #53

Earlier quoted context omitted.

1. TL;DR: OpenAI must verify HumanEval data wasn't used in training in order to compare it? 2. Link in the post you replied to. 3. Subjectivity is fine by me! There's a motte & bailey flavor to it if we combine your comment and this one, c.f. "This is why we use the official numbers."

I think you're assuming that OpenAI is incentivized to benchmark honestly. Like every other company for which a benchmark is a goal, they are not.

[deleted]

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#130

Post by daemonologist »

darkteflon wrote: post #95

Totally out of my depth here but is there any chance that this could be added to something like Ollama and run on a 32gb or 64gb M series Mac in due course? I’m currently running base CodeLlama in this manner but it’s the 7b variant.

Looks like it's available: https://ollama.ai/library/phind-codellama/tags

(I don't have a Mac, so I can't say whether/how well it works.)

Post reply on HN