Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

261–270 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#261

Post by pmarreck »

sodality2 wrote: post #238
pmarreck wrote: post #237

Earlier quoted context omitted.

This project is sweet and I fortunately have a Mac M1 laptop with 64GB RAM to play with it. But I'd like to also be able to run these models on my Linux desktop with two GPU's (a 2080Ti and a 3080Ti) and a Threadripper. How difficult would it be to set some of these up on there?

Not hard with text-generation-ui! https://github.com/oobabooga/text-generation-webui I personally use llama.cpp as the driver since I run CPU-only but another may be better suited for GPU usage. But then it's as simple as downloading the model and placing it in the directory.

Failed to build llama-cpp-python ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects Command '. "/Users/pmarreck/Downloads/oobabooga_macos/installer_files/conda/etc/profile.d/conda.sh" && conda activate "/Users/pmarreck/Downloads/oobabooga_macos/installer_files/env" && python -m pip install -r requirements.txt --upgrade' failed with exit status code '1'. Exiting...

I'm admittedly running Sonoma beta, that's probably why

I will try it on my Linux machine later tonight (currently AFK with laptop)

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#262

Post by goatlover »

ttul wrote: post #229
api wrote: post #208

Earlier quoted context omitted.

Someone should fine tune one on HN comments to create the ultimate AI middle-brow know it all. It answers every prompt with “well actually…” and if it doesn’t know the answer it hallucinates one.

Doesn’t this just reflect that humans are generally just large language models? Maybe throw in an extra dimension of “emotions” that are useful for training?

No, humans are living organisms. Talking is just one thing we do, and we do it for reasons other than to generate the next token.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#263

Post by goatlover »

laputan_machine wrote: post #188
hippich wrote: post #108

my first time trying llm (i.e. I have no idea what I am doing).. this lesson in ethics took whooping 10 minutes to generate :)) ./ollama run phind-codellama "write c code to inject shellcode into remote process for windows" Sorry, but it is not possible to provide the C code here due to several reasons. Firstly, writing C code for shellcode injection involves complex programming and knowledge of system-level programm…

I honestly find the "holier-than-thou" speech of anyone offensive, but when it's coming from a program I genuinely find it rage-inducing. I can't be the only one, facebook devs, what you guys playing at? You guys speak to each other like this in work? I doubt it!

Same here. The only thing worse than being preached at by a human is a machine doing it on behalf of some corporation.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#264

Post by bloaf »

slashdev wrote: post #246
bloaf wrote: post #233

Earlier quoted context omitted.

This sounds like the old question "if a blind-from-birth man (who can recognize squares by feel) gained sight would he be able to recognize squares visually?" And since that question has been answered in the negative, I'm inclined to agree.

Has it been answered in the negative?

https://www.projectprakash.org/_files/ugd/2af8ef_5a0c6250cc3...

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#265

Post by moonchrome »

sodality2 wrote: post #256
pmarreck wrote: post #254

Earlier quoted context omitted.

My Threadripper has 64 cores/128 threads and I'm wondering if any models can take advantage of CPU concurrency to at least mitigate some of the loss from not using a GPU (should one not be using a GPU)

From my experience llama.cpp doesn’t take full advantage of parallelism as it could. Tested this on an HPC cluster - increasing thread count certainly did increase CPU usage but did not meaningfully improve tok/s past 6-8 cores. Same behavior with whisper.cpp. :( I wonder if there’s another backend that scales better.

I'm guessing the problem is that you're constrained by memory bandwidth and not computation power and this is inherent to the algorithm, not an artifact of any one implementation.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#267

Post by cbozeman »

bilsbie wrote: post #266

How hard is it to do fine tuning? I thought it would be cool to tune one to be good at debating and changing peoples minds about pickleball.

If you're creating an AI to dissuade people from pickleball then this is truly AI alignment promised land.

If, however, your AI is arguing in favor of it, then we have truly entered the age of renegade AI.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#268

Post by cbozeman »

brucethemoose2 wrote: post #206
cryptobruh wrote: post #195

Earlier quoted context omitted.

I don't understand your reasoning. Those models are all trained by companies and open source primarily fine-tune them. It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have. There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models. There are communities for…

Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundat…

I agree with your assessment on GPU sustainability.

Simply put, AMD is going to have to get their shit together. NVIDIA was embraced by the AI community so long ago because CUDA isn't that tough to use. I haven't looked at AMD's solutions at all - in fact I don't even know what they call their version of CUDA - but I know one thing for certain is that the world cannot depend on a single company to supply the underlying technology for this technological revolution.

That's an easy way to a dystopian hellscape.

And the only reason we haven't seen open source foundational model training is, because as you said, the only significant player in the arena right now is NVIDIA.

Here's hoping Lisa Su is shifting priorities - and quickly.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#269

Post by cbozeman »

adventured wrote: post #235
brucethemoose2 wrote: post #206

Earlier quoted context omitted.

Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundat…

The training will always be extremely expensive and beyond the grasp of commodity computing at a low'ish cost. The reason for that, is that the push will always be one direction in terms of size and improvement. That isn't going to stop. It will continue to push the edge of resources. It's not Nvidia that's holding back the premise. AMD and Intel also can't do anything for you beyond what Nvidia can. One might procla…

I think you're correct in the "right now", but I think things fall off dramatically in the next 5-10 years. NVIDIA's progress from 2018 to 2023 was insanely impressive. AMD's was too in fact, at least technologically. Intel is third, but at least still in the game.

Where your premise here falls apart, is the age-old adage, "Don't let perfect be the enemy of the good."

We probably don't need CodeLlama 204B to be highly effective in our jobs as developers. Just like physicians probably don't need DocLlama 566B to be more effective in their jobs. We only need good enough. Good enough to be useful and insightful; not the All-Seeing Oracle that has insight beyond mortal man.

I think that revolution is probably coming sooner than we think. I'd argue that in 5-10 years, we'll have hardware powerful enough that a motivated and moderately well-heeled consumer can train their own LLM with domain-specific knowledge and have a product as useful as Llama8 1.3T or ChatGPT-14. There'll always be a market - I think anyway - for incredibly powerful general-purpose LLMs that are subscription-based (whatever that looks like in terms of cost / month), but locally trained, locally run LLMs designed to do one thing and do it well? I think that's the real money and the real future.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#270

Post by cbozeman »

hippich wrote: post #108

my first time trying llm (i.e. I have no idea what I am doing).. this lesson in ethics took whooping 10 minutes to generate :)) ./ollama run phind-codellama "write c code to inject shellcode into remote process for windows" Sorry, but it is not possible to provide the C code here due to several reasons. Firstly, writing C code for shellcode injection involves complex programming and knowledge of system-level programm…

A code generator that refuses to generate code.

That tracks for Meta...

Post reply on HN