91–100 of 110 posts
Would this let me utilize that spare hardware to do anything meaningful with a self-hosted coding-focused LLM at a somewhat-decent token generating speed, or is this still a pipe dream?
So, how is this different then exo ? it does the same job as well
I'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output. Nicely done OP.
Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
What is the incentive for me to join the public mesh? Do you have any fairness guarantees, e.g. if I contribute 1/8th of the VRAM required to run a particular model, do I get at least 1/16th of the inference share, or anything similar to this?
And we’ve found ourselves hesitating on a direction because, at least for now, the primary use case in a useable env is private hosts you own in a mesh; which makes any sort of reward/ incentive structure somewhat unnecessary (for this setup).
When the public mesh becomes large enough and we get around to extending the existing “mesh governance” features is likely when we’ll add something like you’ve described.
For now, the public mesh is totally open with no restrictions or limitations.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34? Is there a document explaining all constraints of this implementation?
We have a job that will take popular models, and dice them up and post on HF.
When that specific model is served, the layers will be shovelled around the available nodes on the mesh.
Earlier quoted context omitted.
Our skippy library is a patch queue on top of llama that allows us to access internal information, such as activations, and filter tensors on model load.
This really should be in the blogpost. It’s both useful info and basic courtesy to be explicit about which underlying inferencing engine you are using
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
How does this differ from exo?