Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

151–160 of 175 posts

Re: OpenLLM

#151

Post by MacsHeadroom »

frodowtf wrote: post #111
leetharris wrote: post #87

Earlier quoted context omitted.

For falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.

Do you know any "standard" way or measures to determine the approximate hardware requirements of a model?

Model size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that.

For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)

Re: OpenLLM

#152

Post by aarnphm »

philoko wrote: post #90
DrNosferatu wrote: post #83

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.

Currently on main, 8bit and 4bit quant is supported

One can simply do

```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```

Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization

Re: OpenLLM

#153

Post by chaoyu »

spencerchubb wrote: post #150

What is the license like for this? Correct me if I'm wrong, but I think the official Llama has a license that allows research use. Would this have a similar restriction if it had the same model architechture but different parameters?

OpenLLM itself is under Apache 2 license, which does NOT restrict commercial use. However, OpenLLM as a framework can be extended to support other LLMs which may come with additional restrictions.

Re: OpenLLM

#154

Post by chaoyu »

xvilka wrote: post #49

Does it work only with text? Or image/video processing too?

Check out BentoML, which is the underlying serving framework used by OpenLLM, and it supports other type of models and modality such as images and videos.

Re: OpenLLM

#155

Post by OkGoDoIt »

bsaul wrote: post #45
aarnphm wrote: post #12

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Side question : why are people working on open source project communicating through discord a lot noawadays ? are discord conversations persisted and indexed on search engines ?

My understanding from asking several people, since I hate discord and want to know why people insist on using it, is that it’s a free alternative to Slack. Simple as that.

But it’s crazy, people are aggressive about Discord for some reason. I maintain an OpenAI SDK package for .Net, and I had some random person decide they wanted it to be a Discord community, so they created a Discord claiming it was the official community discord for my library, and submitted a PR updating my readme to say that it’s my project’s official Discord. They also replied to several issues and pull requests telling people to discuss it on that discord. If Discord isn’t paying this person in some guerrilla marketing tactic, they should be...

Re: OpenLLM

#157

Post by Aerroon »

leetharris wrote: post #87
DrNosferatu wrote: post #83

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

For falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.

To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML

The number after q determines how many bits the weights are. Eg q4 means that is 4-bit.

If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way.

Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.

Re: OpenLLM

#158

Post by PcChip »

Aerroon wrote: post #157
leetharris wrote: post #87

Earlier quoted context omitted.

For falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.

To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger mode…

Let's say I wanted to use one of their quantized models with this OpenLLM project. How would I do that?

Re: OpenLLM

#159

Post by treprinum »

tangjurine wrote: post #140
treprinum wrote: post #52

Earlier quoted context omitted.

Unfortunately, for transformer-based LLMs the magic starts only when they are trained by more that 10^22 TFlops (preferably 10^24) so smaller models might not cut it even for fine-tuned tasks.

Any references on this?

https://www.youtube.com/watch?v=tVtOevLrt5U

Re: OpenLLM

#160

Post by Aerroon »

PcChip wrote: post #158
Aerroon wrote: post #157

Earlier quoted context omitted.

To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger mode…

Let's say I wanted to use one of their quantized models with this OpenLLM project. How would I do that?

Sorry, I don't know. I suspect that it's not possible (yet?). OpenLLM lists a bunch of models in the Github Readme. I think the best way would be to use those for now.
Post reply on HN