21–25 of 25 posts
I’ve been thinking about this same idea recently, so I’m glad it exists now! The biggest benefit I see is to enable RAM constrained GPUs to perform inference of large parameter models with surprisingly high throughput. Because only a single expert is resident, the memory to compute ratio over the network is limited only by the activations, not the weights. For an Moe like kimi k3 where active parameters are 103B, we…
This looks really interesting, and if I understand what this does properly: it was high time someone built this! Without diving into an experiment myself, it would be amazing if you could add some stats or experiment logs, if it's not too much problem and you have them. For example: Given model XYZ, every assuming 5 donors with a uniform 32GB each, each forward pass shunts xGB over the link. Each pass takes nMS, etc.…
Earlier quoted context omitted.
Ho thanks for the comment. Verification does not depend on temperature. Expert execution is deterministic (pure matmul). LUMABRI_VERIFY=N re-runs N% of the calls on a second replica and requires byte-identical output. Temperature (and sampling) happens only on the chatter, after the experts return their activations. So it can be any value (0, 0.7, 1.2…) without affecting the verification contract.
> Expert execution is deterministic (pure matmul). Isn't that only true in theory but wrong in practice due to floating points?
second, don't use vague names of models; point to the actual repos you're testing on huggingface. There's enough diversity and specialization that even if you're smart enough to know that colibri is doing something that's particularly applicable to a type of model, it's easy to get lost in all the acronyms.
third, this looks like a fun tool to unite the diversity of random hardware people have, which is always going to win for local inference.
Earlier quoted context omitted.
Ho thanks for the comment. Verification does not depend on temperature. Expert execution is deterministic (pure matmul). LUMABRI_VERIFY=N re-runs N% of the calls on a second replica and requires byte-identical output. Temperature (and sampling) happens only on the chatter, after the experts return their activations. So it can be any value (0, 0.7, 1.2…) without affecting the verification contract.
> Expert execution is deterministic (pure matmul). Isn't that only true in theory but wrong in practice due to floating points?