Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#131

Post by AnonymousPlanet »

unoti wrote: post #61
AnonymousPlanet wrote: post #52

Earlier quoted context omitted.

Cost isn't the only incentive not to use an LLM service that resides in a foreign country. Around here, there are industries for which it's pretty much a no-brainer to avoid anything that sends data across the atlantic.

Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world. (Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say tha…

Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies?

Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last question, of course, but please think about it. It doesn't matter where the LLM is located but who controls it and who holds the resulting data.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#132

Post by throw03172019 »

Arctic_fly wrote: post #53
idosh wrote: post #45

Can you elaborate on your plans for OpenPipe? Sounds like a very interesting project

Currently OpenPipe allows you to capture input/output from a powerful model and use it to fine-tune a much smaller one, then offers you the option to host through OpenPipe or download it and host it elsewhere. Models hosted on OpenPipe enjoy a few benefits, like data drift detection and automatic reformatting of output to match the original model you trained against (think extraction "function call" responses from a…

Any compliance yet? HIPAA etc

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#133

Post by ttt3ts »

zakki wrote: post #130
ttt3ts wrote: post #124

Earlier quoted context omitted.

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

Would you mind to share all your PC HW (mobo, casing, cooling, etc) for this dual GPU configuration? Thanks.

The one you could build for under 2K is last gen hardware.

* Chenbro Rackmount 4U Server Chassis RM42300-F (rack mount case Remove the air filter on 120mm fan. Put two decent 80mm exhaust at rear). * Two used air cooled 3090s. About $650 a piece on ebay. Check slot width and make sure everything will fit on your motherboard. Do a burn in when you get them cause used GPUs can be hit or miss. * 5950x CPU (overkill just had it) * 128GB DDR4 * Motherboard with x570 chipset and dual pcie x16. These will birificate to x8 pcie 4.0 lanes to each GPU. This is enough bandwidth to push GPUs to max IME * 1200W+ ATX power supply. * ebay "u.2 pcie 3.84TB" and adaptor for m.2 NVME slot. (again what I had & it is cheap)

If you're going to really beat the thing I would power limit the 3090s to 320w (from 350w). Perf change is not really notable and keeps temps better.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#134

Post by refulgentis »

homarp wrote: post #89
refulgentis wrote: post #33

Earlier quoted context omitted.

For use cases well within the capabilities of an LLM from last year, fine-tuned LLaMa 2 13B should/will blow ChatGPT out of the water: think "rate the sentiment of this text from 0-10". I believe this because LLaMa-2 13B is more than good enough to handle what I call "quick search", i.e. ``` User: "What's the weather in Milwaukee?" System: Here's some docs, answer concisely in one sentence. AI: It's 73 degrees Farenh…

you already have TVM for the cross platform stuff see https://tvm.apache.org/docs/how_to/deploy/android.html or https://octoml.ai/blog/using-swift-and-apache-tvm-to-develop... or https://github.com/mlc-ai/mlc-llm

My deepest thanks, I owe you one. Overlooked this completely. & spent dozens of hours learning way too much to still fall short of understanding how to make it work in ONNX.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#135

Post by zakki »

ttt3ts wrote: post #133
zakki wrote: post #130

Earlier quoted context omitted.

Would you mind to share all your PC HW (mobo, casing, cooling, etc) for this dual GPU configuration? Thanks.

The one you could build for under 2K is last gen hardware. * Chenbro Rackmount 4U Server Chassis RM42300-F (rack mount case Remove the air filter on 120mm fan. Put two decent 80mm exhaust at rear). * Two used air cooled 3090s. About $650 a piece on ebay. Check slot width and make sure everything will fit on your motherboard. Do a burn in when you get them cause used GPUs can be hit or miss. * 5950x CPU (overkill just…

Thank you so much.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#136

Post by pdntspa »

thewataccount wrote: post #82
ramesh31 wrote: post #56

Earlier quoted context omitted.

>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT. Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.

What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

I bought an old server off ServerMonkey for like $700 with a stupid amount of RAM and CPUs and it runs Llama2-70b fine, if a little slowly. Good for experimenting

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#138

Post by carom »

AnonymousPlanet wrote: post #131
unoti wrote: post #61

Earlier quoted context omitted.

Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world. (Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say tha…

Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies? Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last questi…

I do think large tech companies do pretty well with customer data. As a former Googler I would be comfortable with my Gmail data residing in a foreign datacenter.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#139

Post by apstls »

ttt3ts wrote: post #124
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

Are there any good resources related to expanding context windows, or even just the mechanics of how they actually work as properties of a model?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#140

Post by ttt3ts »

apstls wrote: post #139
ttt3ts wrote: post #124

Earlier quoted context omitted.

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

Are there any good resources related to expanding context windows, or even just the mechanics of how they actually work as properties of a model?

Lots. LLAMA 2 was trained on 4K context windows but can run on arbitrary length just the results become garbage as you go longer.

I refer you to https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-c... for an "easy" to digest summary

Post reply on HN