Mesh LLM: distributed AI computing on iroh
iroh.computer
Mesh LLM: distributed AI computing on iroh
1–10 of 110 posts
Re: Mesh LLM: distributed AI computing on iroh
#2It sounds like iroh enables distributed compute without having to finangle custom hardware.
Re: Mesh LLM: distributed AI computing on iroh
#3I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
Re: Mesh LLM: distributed AI computing on iroh
#4[flagged]
Re: Mesh LLM: distributed AI computing on iroh
#5cocompute.ai is already doing this really well.
Re: Mesh LLM: distributed AI computing on iroh
#6cocompute.ai is already doing this really well.
Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
Re: Mesh LLM: distributed AI computing on iroh
#7cocompute.ai is already doing this really well.
Cool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.
Re: Mesh LLM: distributed AI computing on iroh
#8I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
Re: Mesh LLM: distributed AI computing on iroh
#9cocompute.ai is already doing this really well.
Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
Good point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
Re: Mesh LLM: distributed AI computing on iroh
#10Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?