Viewing profile — junrushao1994

This is great! Have you guys considered integrating with one of the existing systems?

Yeah thanks for sharing! This is definitely super valuable data and insights :) Regarding exllama-V2, MLC/TVM does benchmark against it: - Single GPU: https://github.com/mlc-ai/llm…

Machine Learning Compilation (MLC) now supports compiling LLMs to multiple GPUs. For Llama2-70B, it runs 4-bit quantized Llama2-70B at: - 34.5 tok/sec on two NVIDIA RTX 4090 at $3k…

Ah please help us by submitting a PR! I noticed the rust build failed last night but didn’t get a chance to look into it

This is a particular unique course offering introduction on ML compilation and deployment :)

As of today performance in WebGPU isn't as competitive yet, but there are really quite a lot of low-hanging fruits for WebGPU to pick up.

That's a great idea! We should dig around and see if there's any plugin to use

This is amazing to hear Steven! (Sorry I locked myself out of discord a couple of days ago...) I'm sure there's bunch of features missing like biased sampling you mentioned, and mo…

True and there are some other issues to be addressed. Those two particular issue is on our roadmap. Regarding quantization, we wanted to develop a code path that absorbs any quanti…

LLM decoding is dominated by memory bandwidth, and 3090Ti and 4090 happen to have the identical theoretical memory bandwidth

We haven't done any comparison them yet, but generally we believe Vulkan as a more generic cross-vendor API should be slower than ROCm. Same for CUDA vs Vulkan.

Well, I'm very much into true open source, and my belief is that any contributor is automatically part of the team :)

Generally speaking I expect Vulkan to be slower than ROCm given it's designed for generic gaming across GPU vendors, so the takeaway is, whenever ROCm is available and usable, we s…

> Can you comment on how difficult it was to achieve this, and what the relative advantages b/w cards? Thanks for asking! I personally believe TVM Unity is a proper software stack …

Really depends on how good ROCm support for WSL2 is. Our team don't have a windows machine so could not verify ourselves, but if you got ROCm set up properly on WSL2, MLC LLM shoul…

ROCm has improved a lot over the past few months, and now ROCm 5.6 seems to work out of box by just following this tutorial: https://rocm.docs.amd.com/en/latest/deploy/linux/instal…

yeah we tried out popular solutions like exllama and llama.cpp among others that support inference of 4bit quantized models

tbh im not sure what amds plan is on ROCm support on consumer devices, but i dont really think amd is being fraudulent or something. Both rocm and vulkan are supported in MLC LLM a…

One of the authors here. Glad it’s on HackerNews! There are two points I personally wanted to make through this project: 1) With a sufficiently optimized software stack, AMD GPUs c…

I don't think TVM advertised a lot on its full capabilities, for example, high-perf codegen for dynamic shapes without auto-tuning, or auto-tuning-based codegen, at least in the pa…

Yeah I believe countless new research and product ideas will be built on top of the open-source Llama-2