Viewing profile — paul_mk1

Sub 1-bit has been done at least as far back as 2016 for VGG style networks (my work). I was able to get 0.68 "effective" bits. The idea is that in each forward pass you add noise …

Goes back before then. This got popularized by BinaryConnect in 2015, and groups were training binary networks as early as 2011. You are probably referring to XNOR net, and the nov…

It's clear that QLoRA has opened up finetuning to a wider audience with limited compute, which is a good thing. One thing I've wondered about: what are the drawbacks to using QLoRA…

I don't think there is anything conceptually new in this work, other than it is applied to LLMs. But in fairness, getting these techniques to work at scale is no small feat. In my …

Nice to know there is a trail to relevant citations. I missed the BitNet paper and need to catch up. Btw TrueNorth project evolved into "NorthPole" chip by the same group, and was …

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects t…

Comparisons between different chip architectures are imperfect. In our opinion the most fair thing to do is 1) match the TFLOPs (since these workloads are compute bound), and 2) fi…

You can try it yourself on SageMaker using NVIDIA. There's a free trial. https://aws.amazon.com/marketplace/seller-profile?id=seller-... For AMD, you'll have to wait until these ca…

This is a good observation, the cards do have different memory bandwidth with the MI210 having more than double the bandwidth via HBM2e. Note that the comparisons between the two c…

>The "-ngl 32" means that only 32 out of 35 layers are being run on the GPU, and this results in a huge slow down as the GPU syncs with the CPU, and then computes the last 3 layers…

Appreciate your response. We compared MKML mk600 (5.2GB) against llama.cpp Q5_1 (4.7GB) and Q6_k (5.1GB) on a 4090 for llama-7B. The test is the same in all cases: we generate 128 …

Hi, one of the founders here. Attempting to address some of the comments in a single message. To help understand why we decided not to compare to existing methods: I think it would…