Punica: Serving multiple LoRA finetuned LLM as one
1–10 of 28 posts
Re: Punica: Serving multiple LoRA finetuned LLM as one
#2This is great! Have you guys considered integrating with one of the existing systems?
Re: Punica: Serving multiple LoRA finetuned LLM as one
#3Good job!
I observed that you implemented many cuda kernels by yourselves. Just wondering your consideration or trade-off between implementating the kernels via pure CUDA code vs. implementing based on compiler like TVM/Triton.
Re: Punica: Serving multiple LoRA finetuned LLM as one
#4Good job! I observed that you implemented many cuda kernels by yourselves. Just wondering your consideration or trade-off between implementating the kernels via pure CUDA code vs. implementing based on compiler like TVM/Triton.
Good question, in general implementing kernels on page tables is tricky in Tensor Compilers because integer set analysis might fail sometimes (but can be fixed with some tweaks). I think using compilers like TVM can help deploy serving systems on different platforms (e.g. AMD GPUs) and I'm optimistic about this direction (and we have to make Tensor Compilers more user-friendly).
Re: Punica: Serving multiple LoRA finetuned LLM as one
#5This is great! Have you guys considered integrating with one of the existing systems?
Thanks for the question. Currently Punica is on the ecosystem of PyTorch and HuggingFace Transformers. So PyTorch users can start to use Punica now.
Look forward to collaboration with TVM and MLC to reach more users :)
Re: Punica: Serving multiple LoRA finetuned LLM as one
#6Nice!
Any thoughts as to how this would come together with serving frameworks like vLLM, lmdeploy, Triton Inference Server, etc?
Re: Punica: Serving multiple LoRA finetuned LLM as one
#7Awesome work! Here's a recent paper released yesterday, also focused on efficiently serving many LoRAs simultaneously: https://arxiv.org/abs/2311.03285
Really looking forward to these innovations becoming more widespread -- I expect we're very close to a world where training a LoRA on a one-off task like "review every HN post from the last 3 years and flag any of them that contain informed speculation about the architecture of GPT-4" will be easy, cheap and routine.
Re: Punica: Serving multiple LoRA finetuned LLM as one
#8Great work! I am curious that how much effort it would take to support LoRAs with different ranks?
Re: Punica: Serving multiple LoRA finetuned LLM as one
#9Am I correct in understanding that LoRA is basically a way to cheaply create “delta” LLMs that apply onto the main large one to create a specialization? In other words, this would obviate all the vector DB stuff that people are doing right?
Re: Punica: Serving multiple LoRA finetuned LLM as one
#10Am I correct in understanding that LoRA is basically a way to cheaply create “delta” LLMs that apply onto the main large one to create a specialization? In other words, this would obviate all the vector DB stuff that people are doing right?
The general consensus imo is that fine-tuning is more for tone and style vs accuracy. People use vector DBs to grab relevant data to throw into the prompt and call it Retrieval Augmented Generation.
From what this seems to do is host multiple deltas fine-tunings and hot swap as needed. Incredible optimization. It's like going from AMIs to ECS or Kubernetes.