Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mesh LLM: distributed AI computing on iroh

iroh.computer

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Mesh LLM: distributed AI computing on iroh

#81

Post by maccam912 »

I have a macbook pro, figured I'd see how easy it was to contribute some vram...

And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc.

It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!

Re: Mesh LLM: distributed AI computing on iroh

#82

Post by chromatin »

stymaar wrote: post #58
SwellJoe wrote: post #8

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…

> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…

A 1 gbe local network should have thus network latency should not be the limiting factor in reality, no?

Re: Mesh LLM: distributed AI computing on iroh

#84

Post by postpress »

This got me thinking about experiments with models talking to each other over WebRTC: https://xt-ml.github.io/shadow-claw/

Its sort of a "P2P mesh" :-) Watch four instances of the harness running together and collaborating on checking the weather: https://www.youtube.com/watch?v=h1les1A3gcg

Re: Mesh LLM: distributed AI computing on iroh

#85

Post by jjheuaao »

maccam912 wrote: post #81

I have a macbook pro, figured I'd see how easy it was to contribute some vram... And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I co…

[flagged]

Re: Mesh LLM: distributed AI computing on iroh

#86

Post by throw1234567891 »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline.

Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?

Is there a document explaining all constraints of this implementation?

Re: Mesh LLM: distributed AI computing on iroh

#87

Post by stymaar »

chromatin wrote: post #82
stymaar wrote: post #58

Earlier quoted context omitted.

> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…

A 1 gbe local network should have thus network latency should not be the limiting factor in reality, no?

On a fast network, yes. If you want to create a distributed network over the internet, then it will.

Re: Mesh LLM: distributed AI computing on iroh

#88

Post by rao-v »

i386 wrote: post #68
stymaar wrote: post #60

Earlier quoted context omitted.

Is it a fully custom inference engine or are you reusing parts of an existing stack? (llama.CPP, vLLM, etc.)

Our skippy library is a patch queue on top of llama that allows us to access internal information, such as activations, and filter tensors on model load.

This really should be in the blogpost. It’s both useful info and basic courtesy to be explicit about which underlying inferencing engine you are using

Re: Mesh LLM: distributed AI computing on iroh

#89

Post by drawnwren »

khalic wrote: post #67
drawnwren wrote: post #49

Earlier quoted context omitted.

It's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.

M5 max has 614GB/s, you mean the m4?

Oh, I was looking at every M5 except for the 40-core M5 Max. They have 460.

Re: Mesh LLM: distributed AI computing on iroh

#90

Post by derdi »

vigsterkr wrote: post #70

the https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.

They might want to make their docs less confusing. https://query.mt/showcase/gpu-sharing/ says "no API key needed", and talks about running llama.cpp on a friend's GPU. No API keys are needed for this. But then the page goes on about who "holds" API keys, and where API keys "stay", where API keys are "locked", etc.
Post reply on HN