Universal LLM Deployment Engine with ML Compilation
1–8 of 8 posts
Re: Universal LLM Deployment Engine with ML Compilation
#2A unified efficient open-source LLM deployment engine for both cloud server and local use cases.
It comes with full OpenAI-compatible API that runs directly with Python, iOS, Android, browsers. Supporting deploying latest large language models such as Qwen2, Phi3, and more.
Re: Universal LLM Deployment Engine with ML Compilation
#3runs on qwen2 on iphone with 26 tok/sec and a OpenAI style swift API
Re: Universal LLM Deployment Engine with ML Compilation
#4From first-hand experience, the all-in-one framework really helps reduce engineering effort!
Re: Universal LLM Deployment Engine with ML Compilation
#5AI ALL IN ONE! Super universal and performant!
Re: Universal LLM Deployment Engine with ML Compilation
#6The MLCEngine presents an approach to universal LLM deployment, glad to know it works for both local servers and cloud devices with competitive performance. Looking forward to exploring it further!
Re: Universal LLM Deployment Engine with ML Compilation
#7Looks cool. I'm looking forward to trying building some interesting apps using the SDKs.
Re: Universal LLM Deployment Engine with ML Compilation
#8Glad to see MLC is becoming more mature :) I can imagine the unified engine could help build agents on multiple devices.
Any ideas on how those edge and cloud models collaborate on compound tasks (e.g. the compound ai systems: https://bair.berkeley.edu/blog/2024/02/18/compound-ai-system...)