Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#141

Post by skybrian »

NavinF wrote: post #48
tikkun wrote: post #3

Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls. My other thoughts to extend this are that you could make it…

>GPT and other LLMs don't allow you to use their output to train competing models ToS is unenforceable and irrelevant to anyone that's in this space

That seems mostly right, particularly for internal models, but I wonder about adding some ringers to prove that copying happened:

https://en.m.wikipedia.org/wiki/Trap_street

Also, it seems sort of like how cryptocurrency folks assumed their transactions were anonymous? It's an API, so they could log the calls. (Maybe not the contents.)

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#142

Post by deet »

unoti wrote: post #61
AnonymousPlanet wrote: post #52

Earlier quoted context omitted.

Cost isn't the only incentive not to use an LLM service that resides in a foreign country. Around here, there are industries for which it's pretty much a no-brainer to avoid anything that sends data across the atlantic.

Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world. (Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say tha…

Azure GPT 4 is already available in: Australia East, Canada East, East US, East US 2, France Central, Japan East, Sweden Central, Switzerland North, UK South (https://learn.microsoft.com/en-us/azure/ai-services/openai/c...)

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#143

Post by fomine3 »

AnonymousPlanet wrote: post #131
unoti wrote: post #61

Earlier quoted context omitted.

Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world. (Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say tha…

Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies? Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last questi…

Worldwide big corps already utilized Microsoft 365 especially SharePoint. That's Microsoft's advantage.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#144

Post by jpdus »

ramesh31 wrote: post #68
YetAnotherNick wrote: post #67

Earlier quoted context omitted.

Which model? I am talking about 70b as mentioned clearly. 70b 8b is 70GB just for the model itself. How much token/second are you getting with single 4090?

Offloading 40% of layers to CPU, about 50t/s with 16 threads.

Care to share your detailed stack and command to reach 50t/s? I also have a 7950 with DDR 5 and I don't even get 50 t/s on my two RTX 4090s....

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#146

Post by atleastoptimal »

Fine-tuned low parameter LLM's are superficially good but the cracks are obvious if you test them on anything that isn't very strictly tied to the training data. IMO GPT-4 is really the first LLM that's broken out of the fake intelligence quality most LLM's seem to have, though only by a little.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#147

Post by behnamoh »

Arctic_fly wrote: post #47
binarymax wrote: post #36

This looks awesome! Tangential question - do you find GPT function calling to work consistently and without error, or do you get errors when using it? By errors I mostly mean incorrect function signatures/types or missing values...but if you see other unpredictable behavior that would help too.

I haven't had much trouble with GPT 3.5 or 4 function calls returning in an undesirable format recently. I did get a few bad syntax responses when OpenAI first rolled it out, but not for the past few months. Llama 2 can also pick the function call format up, given sufficient training data that contains function call responses, though you'll then have to parse the returned object out of the text-based response.

Has anyone done such fine tuning on llama though? Afaik most projects like llama.cpp use grammars instead.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#148

Post by animeshjain »

tikkun wrote: post #3

Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls. My other thoughts to extend this are that you could make it…

I would be interested in reviewing your handbook too. I am technical, but have not deployed any AI related tooling so far. keen to know if this is targeted to AI noobs as well.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#149

Post by coryrc »

ttt3ts wrote: post #124
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

> I built two such a systems after burning that much in a week on ChatGPT.

What are you doing!?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#150

Post by yessenzhar »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

We provide per token based Llama 2 70B API at Deep Infra, $1/1M tokens, which is 25-50% cheaper than ChatGPT.
Post reply on HN