Viewing profile — crowwork

The goal of the project is to bring open ABI and FFI for machine learning systems. - Stable, minimal C ABI designed for kernels, DSLs, and runtime extensibility. - Zero-copy intero…

Scale LLM serving with programmable cross-engine serving patterns, all in a few lines of Python

XGrammar is an open-source library for efficient, flexible, and portable structured generation. Bring 2x-10x speedup in grammar grammar-guided(JSON and CFG) LLM serving.

Comes with ability to do full structured generation with json schema also a in-browser demo https://chat.webllm.ai/

runs on qwen2 on iphone with 26 tok/sec and a OpenAI style swift API

2b model running at 20tok/sec on iphone, nice potential for future applications

Runs Phi-2 on Samsung S23 with pretty decent speed on Google Chrome browser. LLM on browser on a phone

You can also try out the vulkan backend, which we know should work for windows, although speed might be slower than rocm

Yes, it works out of box and the blog contains a prebuilt python package that you can try out

There is also vulkan support which should be more universal(also included in the post), for example, the post also shows running LLM on a steamdeck APU.

Checkout the latest docs https://mlc.ai/mlc-llm/docs/ MLC started with demos and it evolved lately, with API integrations, documentations into an inference solution that everyone c…

It certainly also involves generating code(e.g. WebGPU, vulkan) that are more akin to traditionally compiler, and more like graph and memory optimization. So indeed more than packa…

There is a conda app that can be installed on macos