Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mesh LLM: distributed AI computing on iroh

iroh.computer

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Mesh LLM: distributed AI computing on iroh

#101

Post by throw1234567891 »

sig_kill wrote: post #97
throw1234567891 wrote: post #86

Earlier quoted context omitted.

> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34? Is there a document explaining all constraints of this implementation?

It depends on the size / structure of the model… but generally you can check out the MeshLLM huggingface org for pre-computed model splits. We have a job that will take popular models, and dice them up and post on HF. When that specific model is served, the layers will be shovelled around the available nodes on the mesh.

Do you have instructions how to dice models myself? Every model in your org is a measly 4bit. I kinda hoped it is as simple as taking a GGUF and throwing at Mesh LLM but I fear it's not going to be that straightforward :)

Re: Mesh LLM: distributed AI computing on iroh

#103

Post by sig_kill »

unrvl22 wrote: post #50
Abishek_Muthian wrote: post #41

I'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output. Nicely done OP.

Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.

This is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM

Re: Mesh LLM: distributed AI computing on iroh

#104

Post by sig_kill »

roger_ wrote: post #78

Does this support Qwen 3.6 (e.g. 27B) and the myriad of llama.cpp options (batch sizes, quantization, etc.)? I'd love to see some performance data.

Yes! I’ve worked on the settings interface between our runtime and llamacpp, these are documented and available via our config.toml file

Re: Mesh LLM: distributed AI computing on iroh

#106

Post by jmercouris »

stymaar wrote: post #62
jmercouris wrote: post #3

I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.

Throughput is not a problem as you just share relatively small vectors (a few kilobytes in size), the key issue is network latency .

I didn't mean the throughput of the network, I meant the throughput of the calculations. Yes, the latency is the issue.

Re: Mesh LLM: distributed AI computing on iroh

#107

Post by dsegovia »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

This is surprisingly similar to what I'm trying to build (at least at the network level), down to the use of iroh. I have not attempted model splitting though, and the angle is slightly different. Major props for this, it seems like a great project. The more we can push local inference, the better for everyone. Interested to see if there's collaboration opportunities. https://github.com/dsegovia90/cocompute

Re: Mesh LLM: distributed AI computing on iroh

#108

Post by michaelneale »

nullc wrote: post #38

Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.

@null not as yet, but did a tiny bit of research before. If i understand it this would help mainly with batching (ie concurrent sessions/users/turns etc) to keep the pipeline more busy (vs a batch of a fixed size going together), which is interesting, but most work so far has been on latency hiding for straight line performance to start.

We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss).

tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!

Post reply on HN