Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#81

Post by ronyfadel »

haxton wrote: post #77
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

It still does a much better job at translation than llama 2 70b even, at 6.7b params

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#82

Post by thewataccount »

ramesh31 wrote: post #56
loudmax wrote: post #41

Earlier quoted context omitted.

You can run the smaller Llama variants on consumer grade hardware, but people typically rent GPUs from the cloud to run the larger variants. It is possible to run even larger variants on a beefy workstation or gaming rig, but the performance on consumer hardware usually makes this impractical. So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.

>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT. Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.

What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#83

Post by computerex »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

Replicate has terrible pricing. Have you tried deepinfra?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#84

Post by ramesh31 »

thewataccount wrote: post #82
ramesh31 wrote: post #56

Earlier quoted context omitted.

>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT. Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.

What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

>What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

That's an exercise left to the reader for now, and is where your value/moat lies.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#85

Post by pocketarc »

ramesh31 wrote: post #68
YetAnotherNick wrote: post #67

Earlier quoted context omitted.

Which model? I am talking about 70b as mentioned clearly. 70b 8b is 70GB just for the model itself. How much token/second are you getting with single 4090?

Offloading 40% of layers to CPU, about 50t/s with 16 threads.

That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#86

Post by two_in_one »

ronyfadel wrote: post #81
haxton wrote: post #77

Earlier quoted context omitted.

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

It still does a much better job at translation than llama 2 70b even, at 6.7b params

If it's MOE that may explain why it's faster and better...

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#87

Post by hereonout2 »

halflings wrote: post #31
7speter wrote: post #22

Earlier quoted context omitted.

When you say it can run on consumer gpus, do you mean pretty much just the 4090/3090 or can it run on lesser cards?

I was able to run the 4bit quantized LLAMA2 7B on a 2070 Super, though latency was so-so. I was surprised by how fast it runs on an M2 MBP + llama.cpp; Way way faster than ChatGPT, and that's not even using the Apple neural engine.

It runs fantastically well on M2 Mac + llama.cpp, such a variety of factors in the Apple hardware making it possible. The ARM fp16 vector intrinsics, the Macbook's AMX co-processor, the unified memory architecture, etc.

It's more than fast enough for my experiments and the laptop doesn't seem to break a sweat.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#89

Post by homarp »

refulgentis wrote: post #33
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

For use cases well within the capabilities of an LLM from last year, fine-tuned LLaMa 2 13B should/will blow ChatGPT out of the water: think "rate the sentiment of this text from 0-10". I believe this because LLaMa-2 13B is more than good enough to handle what I call "quick search", i.e. ``` User: "What's the weather in Milwaukee?" System: Here's some docs, answer concisely in one sentence. AI: It's 73 degrees Farenh…

you already have TVM for the cross platform stuff

see https://tvm.apache.org/docs/how_to/deploy/android.html

or https://octoml.ai/blog/using-swift-and-apache-tvm-to-develop...

or https://github.com/mlc-ai/mlc-llm

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#90

Post by thewataccount »

ramesh31 wrote: post #84
thewataccount wrote: post #82

Earlier quoted context omitted.

What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

>What's the best way to use LLama2-70b without existing infrastructure for orchestrating it? That's an exercise left to the reader for now, and is where your value/moat lies.

> That's an exercise left to the reader for now, and is where your value/moat lies.

Hopefully more on-demand services enter the space. Currently where I am we don't have the resources for any type of self orchestration and our use case is so low/sporadic that we can't simply have a dedicated instance.

Last I saw the current services were rather expensive but I should recheck.

Post reply on HN