Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
Why wouldn't they? It is their best strategy.
I didn't like them that much before, now I do.
Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
81–90 of 306 posts
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#82Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
If Facebook were to implement a language model inside the platform now it would raise a lot of privacy issues. By publishing free to use models they can rely on others to do the work of training the society into using such tools at mass scale. So by the time they adopt in in all their services there will be much less backlash.
Also, it's cheaper.
Not just for the engineer-hours, but for the endless VP/PM political meetings arguing over this approach or that.
Way cheaper to let the market decide, and then reincorporate the winner into their products. A winner which happens to already be based on a semi-compatible tech stack.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#83Earlier quoted context omitted.
They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.
But that clause pretty much only applies to mega companies like FB no?
Does FB care about any companies smaller than that? If they're Instagram-competitive-successful, FB can afford to buy them.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#84Earlier quoted context omitted.
>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!
I'm aware of at least one study by Stanford. PDF paper linked in this article: https://www.techopedia.com/is-gpt-4-a-flop Of course, I'd like to see more than one study. But this one is by a well known university, and it's pretty conclusive. GPT-4 is getting worse (especially for code, maths, and analytical reasoning) and more censored.
It's important to frame this correctly.
The article is a bit misguided (it doesn't matter which university publishes an article) because there are so many ways in which a model can be altered, even excluding retraining weights. Also, even if the performance has dropped practically due to removing some resources for more shortcuts to be taken (for example changing beam search and typical sampling parameters), making implications about the outlook for the future is not really appropriate, since retraining weights, changing architecture, etc can improve capabilities immensely.
It's important not to suggest that GPT systems in general are on the way outside.ply due to some small alterations in parameters that make a system slightly less performant (which seems to be a popular perspective).
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#85[dead]
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#86Glad to see open source models becoming competitive and thanks for sharing the model to community!
Lepton AI has hosted this model here free for everyone to give it a try (and compare to the original CodeLlama 34B version): https://codellama.lepton.run
API is also available at https://codellama.lepton.run/api/v1 (it's fully compatible with OpenAI's api, so just switching the `api_base` to the this url and all your existing openai client side code should continue working)
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#87Ive used GPT 4 for pretty much all of my programming needs and the convenience of a 20 dollar subscription taking care of everything and letting me use a LLM without having to set up any models or servers has been just so simple, is the 2 percent gain worth looking into running a local model again? I tried running a local model a couple months ago but the perfomance was bad. I know code llama came out very recently b…
For me the use case is asking a local LLM questions about an invention I'm working on, as with ChatGPT I can't be confident the ideas don't make their way into the model. I'm able to run some 13B 5Q models, in my opinion the utility and complexity is somewhere between GPT3 and GPT3.5, which doesn't quite cut it for this purpose. That's not to say anything of the lacking coding abilities. I'm on the fence of getting a 3090. If I do so I think I'll set up a server on the PC so I can query the LLM from my phone just like one can use ChatGPT.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#88Glad to see open source models becoming competitive and thanks for sharing the model to community! Lepton AI has hosted this model here free for everyone to give it a try (and compare to the original CodeLlama 34B version): https://codellama.lepton.run API is also available at https://codellama.lepton.run/api/v1 (it's fully compatible with OpenAI's api, so just switching the `api_base` to the this url and all your ex…
I see you're using [INST] tokens. Please don't do this -- this model was not trained in this format or to be a chat model.
Instead, it should be treated as a completions model (like text-davinci-003) and no system prompt should be provided.
Just tell it what you want.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#89It's amazing to see how rapidly things are moving. You can try out CodeLlama-34B on Anyscale Endpoints (an LLM inference API we're building here at Anyscale for open source LLMs). https://app.endpoints.anyscale.com/
This looks like it might be neat but it has a pretty sparse intro page and then the email signup goes straight into stripe checkout - is there any more info about the service anywhere? Like which models are available, or more pricing info?
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#90Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
They're in the business of encouraging users to post content to get more eyeballs and therefore more ad revenue. So commoditising a way of generating a lot more content is in their interests.