[flagged]
Mesh LLM: distributed AI computing on iroh
71–80 of 110 posts
Re: Mesh LLM: distributed AI computing on iroh
#72[deleted]
Re: Mesh LLM: distributed AI computing on iroh
#73I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
Would this benefit from integrating with the Colibri project announced here just days ago?
Re: Mesh LLM: distributed AI computing on iroh
#74Earlier quoted context omitted.
> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…
Ah, that's interesting. I though there was more data crossing the network. So, why does a DGX Spark come with super fast network if 10Gbps ethernet would be sufficient for splitting a model? I never bought a second Strix Halo on the assumption that the pipe between them would be a limiting factor to using larger models, so obviously there's something I don't understand.
The amount of data is only low for inference, not for training, and AFAIK DGX spark is supposed to be a researcher's machine that can do small-scale training.
Re: Mesh LLM: distributed AI computing on iroh
#75I knew this was possible, i asked chatgpt about a year ago and it said no the latency would be too big of a problem. I spent the best part of a year learning libp2p and was looking for a project to do with it at the time.
Re: Mesh LLM: distributed AI computing on iroh
#76I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…
> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…
I’m staring at this comment for a while now: With 3ms latency combined per token, wouldn’t that mean (1 / latency) = 333 token/s for the theoretical upper bound? I’m not trying to nitpick, just curious if I misunderstand something.
Re: Mesh LLM: distributed AI computing on iroh
#77Earlier quoted context omitted.
> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…
I’m staring at this comment for a while now: With 3ms latency combined per token, wouldn’t that mean (1 / latency) = 333 token/s for the theoretical upper bound? I’m not trying to nitpick, just curious if I misunderstand something.
Indeed, I completely screwed my math up. Looks like 10am is too early in the morning for a Sunday.
33 tps max token generation speed would be for 10ms of network latency in the above example.
Re: Mesh LLM: distributed AI computing on iroh
#78Does this support Qwen 3.6 (e.g. 27B) and the myriad of llama.cpp options (batch sizes, quantization, etc.)?
I'd love to see some performance data.
Re: Mesh LLM: distributed AI computing on iroh
#79[dead]
Re: Mesh LLM: distributed AI computing on iroh
#80Is this truly more secure though? The host can still see your data.