Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mesh LLM: distributed AI computing on iroh

iroh.computer

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Mesh LLM: distributed AI computing on iroh

#21

Post by SwellJoe »

i386 wrote: post #16
SwellJoe wrote: post #8

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…

This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN. The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others. I’m currently working on GLM 5.2 on my lab environment with…

That sounds cool, but it's still pretty meaningless without information about what your home lab looks like. A few DGX Sparks wired up with their fancy super fast network is much different than a few laptops on wifi.

Re: Mesh LLM: distributed AI computing on iroh

#22

Post by [deleted] »

SwellJoe wrote: post #8

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…

[deleted]

Re: Mesh LLM: distributed AI computing on iroh

#23

Post by i386 »

zdw wrote: post #19
i386 wrote: post #16

Earlier quoted context omitted.

This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN. The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others. I’m currently working on GLM 5.2 on my lab environment with…

What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?

The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet.

We use a customized Q2 quantization that preserves sensitive tensors at Q8.

To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.

While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.

Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.

Re: Mesh LLM: distributed AI computing on iroh

#24

Post by iotapi322 »

i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

This is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!

Re: Mesh LLM: distributed AI computing on iroh

#25

Post by i386 »

iotapi322 wrote: post #24
i386 wrote: post #13

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

This is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!

Thank you! AMD is a weak spot in our testing right now. If you’re willing to contribute or let us borrow some compute time, drop in on the Discord.

Re: Mesh LLM: distributed AI computing on iroh

#26

Post by darkpicnic »

metadat wrote: post #20
darkpicnic wrote: post #10

Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?

Just wondering, why do you care about encryption in this context?

If payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.

Re: Mesh LLM: distributed AI computing on iroh

#28

Post by wlesieutre »

i386 wrote: post #23
zdw wrote: post #19

Earlier quoted context omitted.

What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?

The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet. We use a customized Q2 quantization that preserves sensitive tensors at Q8. To reduce compute time per layer, we are developing a custom GLM DSA Metal graph. While we are not yet approaching MTP, we plan to…

Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each

Re: Mesh LLM: distributed AI computing on iroh

#29

Post by dwoosley »

I’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow.

What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.

Re: Mesh LLM: distributed AI computing on iroh

#30

Post by oezi »

darkpicnic wrote: post #26
metadat wrote: post #20

Earlier quoted context omitted.

Just wondering, why do you care about encryption in this context?

If payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.

Even if transport is encrypted, the LLM computations will always be clear text, right?
Post reply on HN