Viewing profile — ruihangl

It's good to see that it contains the standard OpenAI API interface in JavaScript which looks very convenient.

A unified efficient open-source LLM deployment engine for both cloud server and local use cases. It comes with full OpenAI-compatible API that runs directly with Python, iOS, Andro…

Great work! I am curious that how much effort it would take to support LoRAs with different ranks?

Purely running in web browser. Generating 6.2 tok/s on Apple M2 Ultra with 64GB of memory.

Upgrading the model is pretty easy. We just need to build the new model locally in the same way we build the current model. This usually takes fewer than 2min. If people want to de…

Thanks for the pointer! As far as we know the WebGPU development on firefox is a bit lagging behind, so we use Chrome and did not develop this project on firefox.

Yes of course. Optimizing and building the model to the format acceptable by ONNX web runtime will getting this in. On the other hand, we also need to enhance our own runtime (for …

Thanks for your interest! Most of the existing stable diffusion demos rely on a server behind to run the image generation. It means you need to host your own GPU server to support …