Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mesh LLM: distributed AI computing on iroh

iroh.computer

Post reply on HN
Print view View original

31–40 of 110 posts

Re: Mesh LLM: distributed AI computing on iroh

#31

Post by Lerc »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

I have never really delved into kv cache implementation, do they run effectively separate caches per layer?

If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.

Is anyone doing best-of-n with a n stage pipeline running each query offset by one?

Re: Mesh LLM: distributed AI computing on iroh

#32

Post by DerivativeBS »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?

Re: Mesh LLM: distributed AI computing on iroh

#33

Post by i386 »

DerivativeBS wrote: post #32
i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?

It can dynamically route. If a machine drops out of split, the topology is recalculated and the request is automatically retried.

Re: Mesh LLM: distributed AI computing on iroh

#34

Post by Creamsicle47 »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again.

A few questions:

1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.

2.) Any safeguards against malicious actors poisoning model activations?

Re: Mesh LLM: distributed AI computing on iroh

#35

Post by i386 »

Lerc wrote: post #31
i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

I have never really delved into kv cache implementation, do they run effectively separate caches per layer? If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries. Is anyone doing best-of-n with a n stage pipeline running each query offset by one?

Each stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.

Re: Mesh LLM: distributed AI computing on iroh

#36

Post by imrehg »

SwellJoe wrote: post #8

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…

That's about the speed I get on a AMD Ryzen AI 9 HX 370 (inside a Framework 13), with Qwen3.6-35B-A3B, so doing the same on that much larger model...

Re: Mesh LLM: distributed AI computing on iroh

#37

Post by i386 »

Creamsicle47 wrote: post #34
i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again. A few questions: 1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed. 2.) Any safeguards against malicious actors poisoning model activations?

To be honest, both are very tough problems we don't have a good answer for yet. If that is something that concerns you, look into building a private mesh with trusted peers.

Re: Mesh LLM: distributed AI computing on iroh

#38

Post by nullc »

Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.

Re: Mesh LLM: distributed AI computing on iroh

#39

Post by zmmmmm »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?
Post reply on HN