Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Punica: Serving multiple LoRA finetuned LLM as one

github.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Punica: Serving multiple LoRA finetuned LLM as one

#11

Post by [deleted] »

lamroger wrote: post #10
vlovich123 wrote: post #9

Am I correct in understanding that LoRA is basically a way to cheaply create “delta” LLMs that apply onto the main large one to create a specialization? In other words, this would obviate all the vector DB stuff that people are doing right?

The general consensus imo is that fine-tuning is more for tone and style vs accuracy. People use vector DBs to grab relevant data to throw into the prompt and call it Retrieval Augmented Generation. From what this seems to do is host multiple deltas fine-tunings and hot swap as needed. Incredible optimization. It's like going from AMIs to ECS or Kubernetes.

[deleted]

Re: Punica: Serving multiple LoRA finetuned LLM as one

#12

Post by abcdabcd987 »

kcorbitt wrote: post #7

Awesome work! Here's a recent paper released yesterday, also focused on efficiently serving many LoRAs simultaneously: https://arxiv.org/abs/2311.03285 Really looking forward to these innovations becoming more widespread -- I expect we're very close to a world where training a LoRA on a one-off task like "review every HN post from the last 3 years and flag any of them that contain informed speculation about the archi…

Thank you! We are also very excited about combining the fast fine-tuning and efficient serving. In fact, what you just said is very related to one of our very first motivations. In my previous blog post [1], I call this scheme "Just-in-time Fine-tuning". Our previous measurement is that, for a medium-sized webpage (~10K tokens), it takes around 30 seconds to 2 minutes to finetune a LoRA model. Another good side of this JIT fine-tuning scheme is that, we can turn any model into a long-context model.

We'll keep doing more research on finetuning. And hopefully, we'll see the results soon.

[1] https://le.qun.ch/en/blog/2023/09/11/multi-lora-potentials/

Re: Punica: Serving multiple LoRA finetuned LLM as one

#13

Post by abcdabcd987 »

kkielhofner wrote: post #6

Nice! Any thoughts as to how this would come together with serving frameworks like vLLM, lmdeploy, Triton Inference Server, etc?

Certainly! We'd like our good designs to be picked up by frameworks and serve all users. Currently, Punica is built on top of PyTorch and HuggingFace Transformers ecosystems. Therefore, vLLM and LMDeploy, which are also in the PyTorch ecosystem, should have a smooth adaption. As for Nvidia Triton and TensorRT-LLM, since our kernels are written in CUDA, I believe it will also work seamlessly.

We call for the open source community to help us integrate Punica with all frameworks, thus the whole society can benefit from the efficiency improvement!

Re: Punica: Serving multiple LoRA finetuned LLM as one

#14

Post by huac »

I think this is one of the most important possible works for open source LLM's, really glad y'all pushed this forward!

That's not hyperbole. Why is OpenAI able to charge so little for their API's? I have heard rival mega LLM company CEO's complain that OpenAI's prices would be a loss for their rivals. But I think it's still positive margin, and that they can charge low prices for API because they've invested more into managing the infra, sure, but most importantly because they have the best utilization of their existing hardware.

If it costs everyone $X/gpu/hr to serve models, the company that has the most throughput wins on price. In a world without finetunes, the most capable model, the one that can zero- or few-shot the most tasks will have the most usage. Finetuned open models can reach parity with GPT on narrow tasks, but until now, having public providers serve the models was expensive. Your private finetune is only going to be queried by you, not everyone, so it's super expensive to serve on a per token level. With hot swappable LoRA adapters, that calculus changes, and the cost per token can go way down. Super, super exciting!

Re: Punica: Serving multiple LoRA finetuned LLM as one

#15

Post by Palmik »

This is amazing, and will unlock many possibilities. I just recently read the S-LoRA paper, which is related, but it's even better to have a working (and extremely efficient!) implementation.

How hard would it be to adapt your kernels to work with the new-gen quants like AWQ or EXL2?

Re: Punica: Serving multiple LoRA finetuned LLM as one

#16

Post by ghotli »

huac wrote: post #14

I think this is one of the most important possible works for open source LLM's, really glad y'all pushed this forward! That's not hyperbole. Why is OpenAI able to charge so little for their API's? I have heard rival mega LLM company CEO's complain that OpenAI's prices would be a loss for their rivals. But I think it's still positive margin, and that they can charge low prices for API because they've invested more int…

Interesting. I'm not so sure I really 'got' that part of finetunes / LoRA adapters before reading this comment. Makes me want to make one to take it for a spin, see what comes out the other side.

Re: Punica: Serving multiple LoRA finetuned LLM as one

#17

Post by abcdabcd987 »

Palmik wrote: post #15

This is amazing, and will unlock many possibilities. I just recently read the S-LoRA paper, which is related, but it's even better to have a working (and extremely efficient!) implementation. How hard would it be to adapt your kernels to work with the new-gen quants like AWQ or EXL2?

Thanks for your encouragement! We are working on quantization as well. We recently submitted a paper, Atom [1], that uses 4-bit quantization, delivering 7.73x throughput compared to FP16 and 2.53x compared to INT8. Atom is able to maintain a perplexity (i.e., model accuracy) close to FP16, outperforming existing quantization approaches.

We are polishing the 4-bit code. It will be added to Punica code base soon. Please stay tuned :)

[1] https://arxiv.org/abs/2310.19102

Re: Punica: Serving multiple LoRA finetuned LLM as one

#18

Post by zhye »

ruihangl wrote: post #8

Great work! I am curious that how much effort it would take to support LoRAs with different ranks?

It will take some effort to implement operators but not too much (cutlass's group gemm already support different mnk's), however the performance benefit is marginal compared to padding all LoRA ranks to the same rank because all these kernels are not compute bound.

Re: Punica: Serving multiple LoRA finetuned LLM as one

#19

Post by Palmik »

abcdabcd987 wrote: post #17
Palmik wrote: post #15

This is amazing, and will unlock many possibilities. I just recently read the S-LoRA paper, which is related, but it's even better to have a working (and extremely efficient!) implementation. How hard would it be to adapt your kernels to work with the new-gen quants like AWQ or EXL2?

Thanks for your encouragement! We are working on quantization as well. We recently submitted a paper, Atom [1], that uses 4-bit quantization, delivering 7.73x throughput compared to FP16 and 2.53x compared to INT8. Atom is able to maintain a perplexity (i.e., model accuracy) close to FP16, outperforming existing quantization approaches. We are polishing the 4-bit code. It will be added to Punica code base soon. Pleas…

Added to my reading list! The world of quantizations is moving so fast even TheBloke might not be able to keep up!

So Atom base models would be compatible with Punica?

I also wonder, many people already train LoRAs in 8 or even 4 bit (for the base model), would it make sense to match the quantization algo used during training and inference?

Re: Punica: Serving multiple LoRA finetuned LLM as one

#20

Post by therealpygon »

huac wrote: post #14

I think this is one of the most important possible works for open source LLM's, really glad y'all pushed this forward! That's not hyperbole. Why is OpenAI able to charge so little for their API's? I have heard rival mega LLM company CEO's complain that OpenAI's prices would be a loss for their rivals. But I think it's still positive margin, and that they can charge low prices for API because they've invested more int…

Doesn’t OpenAI still operate at significant losses by using massive infusions of capital from Microsoft and other investors? If you are giving away half your product, it’s not surprising that they would be undercutting competition. Not a new strategy.

Underprice to avoid or drive out competition and encourage lock-in, then increase prices when you no longer have competitors or your user base is large enough and reliant enough that your attrition is manageable. Then you sell to a bigger company who grinds it up and integrates into their own products. Same as always. Bonus points if you claim to be open source for the free marketing and/or free development/testing in the form of user contributions before switching to a proprietary model.

Shouldn’t we have a standardized corporate strategy bingo card by now?

Post reply on HN