[flagged]
Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
111–120 of 306 posts
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#112Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
One possible explanation is to make PyTorch victory over TensorFlow/JAX even more apparent: with Google ceasing to release interesting open-source models, and Facebook / Meta continuing doing so (even if under a less permissive license than usual), it means that there's nothing interesting in the TensorFlow/JAX ecosystem, and a lot in the PyTorch one. (context: I work for Google DeepMind, opinions are my own)
Hmm, but I feel like Jax/Pytorch interop is only getting better. Tensorflow abstraction has already firmly lost.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#113Earlier quoted context omitted.
Facebook has had an AI research lab for the better half of a decade now. Their open release model worked well in the past and it seems to be working well with Llama too. That looks like the extent of it. We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.
Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.
The open research tradition has been open for many defining technology, most companies continued releasing for free even when it became clear this tech would be transformative (I think it was obvious by GPT-2).
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#114How to use this or gpt 4 for day to day code? I want to feed my code repo into it and generate/fix/explain code. Is the context windows size going to block me. Has anyone been doing it, with llama 2 models ?
https://cursor.sh is the best I've seen so far. It uses GPT-4. I'm sure someone will hook it up to code llama soon.
Thanks. Is there any open source hosted solution like cursor that can use the Llama models ?
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#115What kind of hardware does it take to fine tune such a model? I've been day dreaming about self-play with these models. Generate 100 leetcode problems. One variant writes test cases, one variant writes code. Make the loss for the tester high if there are bugs (maybe compare to using a fuzzer on the program or have multiple tester variants generating large test suites). The coder variant has loss related to number of failed test cases or retries or something.
Then, just keep running that in a loop, generating, testing, developing.
Is it possible to self-train like this, or attempt to do so, on commodity hardware? Could I make a meaningful change to 34B sized models if I had one or two 4090s? (Or something for a similar, few thousand dollar, price point).
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#116Earlier quoted context omitted.
>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!
It takes like 2-3 experiences of receiving a confidently wrong answer to downgrade your usage. If you use a refactoring tool to rename and it misses one, you won’t use it again.
When I was doing a lot of C++ gamedev, we were definitely doing a lot of stuff that would trip up static analysis, e.g. X-macros.
We would still use refactoring tools even though they would often miss stuff. You just rely on a combination of refactoring tool / search and replace / the compiler.
We would also debug our code in release mode with symbols. You get used to a debugging environment where you don't trust anything you're seeing in variables, etc. too.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#117Earlier quoted context omitted.
Nope, just ask your question. It's not chat-tuned the same way as CodeLlama34B Instruct.
Interesting. That seems a pretty odd choice. Without a prompt format (like almost all instruction finetunes have) the model might continue your instruction instead of answering the instruction, since it does not have a delimiter for where the instruction ends and the response begins.
This is an early experiment that we did in just a few hours. We'll have much better models in just a few days, stay tuned :)
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#118The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…
Increased output leads to increased demand. If last week I could pump out 3 solid PRs that fixed issues in our product, then it will be expected that I pump out 20 PRs using AI in a week. Information Engineers are going to be in more demand than ever, the title is just going to change to Platform Engineering because that's the level we will be expected to operate at with the new tooling. This is why there's a multitude of LLM frontends being released every week. That is no longer skilled labor and an elementary school child with enough time can do it. But building an entire platform that can scale and has 5 9's uptime?
When an AI can do that, then we panic.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#119It's amazing to see how rapidly things are moving. You can try out CodeLlama-34B on Anyscale Endpoints (an LLM inference API we're building here at Anyscale for open source LLMs). https://app.endpoints.anyscale.com/
This looks like it might be neat but it has a pretty sparse intro page and then the email signup goes straight into stripe checkout - is there any more info about the service anywhere? Like which models are available, or more pricing info?
I had the same question. Being thrown into a payment page right after providing your email is very jarring. I don't want to pay for a product without even giving it a spin.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#120Earlier quoted context omitted.
One possible explanation is to make PyTorch victory over TensorFlow/JAX even more apparent: with Google ceasing to release interesting open-source models, and Facebook / Meta continuing doing so (even if under a less permissive license than usual), it means that there's nothing interesting in the TensorFlow/JAX ecosystem, and a lot in the PyTorch one. (context: I work for Google DeepMind, opinions are my own)
Another possible reason might be to attract and retain talent? Speculating wildly but I'd guess some top researchers would value having their work be the industry standard rather than internal.
Your speculation is pretty accurate afaik. Mark commented that releasing open source models is what is needed to retain the best so it is what they will do (paraphrasing) on his interview with Lex Friedman.