Someone needs to make an LLM purpose-built for creating high-quality datasets for fine-tuning other LLMs.
[deleted]
Fine-tune your own Llama 2 to replace GPT-3.5/4
151–160 of 194 posts
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#152Earlier quoted context omitted.
I haven't had much trouble with GPT 3.5 or 4 function calls returning in an undesirable format recently. I did get a few bad syntax responses when OpenAI first rolled it out, but not for the past few months. Llama 2 can also pick the function call format up, given sufficient training data that contains function call responses, though you'll then have to parse the returned object out of the text-based response.
Has anyone done such fine tuning on llama though? Afaik most projects like llama.cpp use grammars instead.
Yep! The linked notebook includes an example of exactly that (fine-tuning a 7b model to match the syntax of GPT-4 function call responses): https://github.com/OpenPipe/OpenPipe/blob/main/examples/clas...
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#153For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…
We provide per token based Llama 2 70B API at Deep Infra, $1/1M tokens, which is 25-50% cheaper than ChatGPT.
Can you provide a larger context length? Looking for a replacement of GPT-3.5 16k model. Might be interested for a higher-scale project.
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#154Genuinely informative reply for those (few) of us on HN who don’t know the details of LLMs, thanks
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#155Earlier quoted context omitted.
Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one exception being for organizations where data security is non-negotiable and they really need to host on-prem. The calculus changes drastically though when you bring fine-tuning in, which lets a much smaller model outperform a larger one on many cla…
Founder of Replicate here. It's early indeed. OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B. Running a fine-tuned GPT-3.5 is surprisingly expensive. That's where using Llama makes a ton of sense. Once we’ve optimized inference, it’ll be much cheaper to run a fine-tuned Llama.
We're working on LLM Engine (https://llm-engine.scale.com) at Scale, which is our open source, self-hostable framework for open source LLM inference and fine-tuning. We have similar findings to Replicate: Llama 2 70B can be comparable to GPT 3.5 price, etc. Would be great to discuss this further!
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#156Earlier quoted context omitted.
Depending on what you're trying to accomplish, I'd highly recommend trying the 7B and 13B models first before jumping to the 70B. They're quite capable and I think lots of folks assume they need to jump to a 70B model when really a smaller one would work fine. That said, you should be able to fine-tune a 70B model on an A100 using QLoRA. However, depending on the specifics of your dataset it might actually be cheaper…
The plan was to do it in-house. And buying 8xA100 is a bit too much ;)
I'm in the exactly same boat. Targeting to fine tune llama 2 70b on 2xA100, with the hope of having one A100 run an 8bit quantized 70b model 24/7.
If you have an experiences to share, successes or failures, please do.
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#157Fine-tuned low parameter LLM's are superficially good but the cracks are obvious if you test them on anything that isn't very strictly tied to the training data. IMO GPT-4 is really the first LLM that's broken out of the fake intelligence quality most LLM's seem to have, though only by a little.
If we assume this is true:
https://iv.nboeck.de/watch?v=K5iDUZPx60E&t=2989
Then there isn't anything in particular which makes their model(s) stand out. On the contrary, they seem rather inefficient, which is probably reflected on the inference cost this gargantuan conglomerate takes to run.
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#158I have such use case: I have Java project I develop, I also used phind-codellama-36B-q8 with very satisfying results to aid the development.
Can I train it further using the project source to let the model "understand" the project context more?
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#159I think the cost calculation here does not reflect the actual scenario where most people face. In real world scenario, we don't get inputs queued up to millions and wait for the GPU to inference them continuously at 100% utilization. We need to ensure the user get their response in time, and assume that we get all the inputs spread out evenly within a month, we have to look at the cost of running GPU for a month vs using OpenAI API.
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#160Earlier quoted context omitted.
Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world. (Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say tha…
Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies? Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last questi…
I don't think this is a promise Microsoft can make. The US Cloud Act states that Microsoft falls under US jurisdiction and it's legally bound to share foreign data if asked by US law enforcement.
"The CLOUD Act asserts that U.S. data and communication companies must provide stored data for a customer or subscriber on any server they own and operate when requested by warrant, but provides mechanisms for the companies or the courts to reject or challenge these if they believe the request violates the privacy rights of the foreign country the data is stored in."