61–70 of 110 posts
Is there a catch? If not, this would be super useful.
It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).
I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…
> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…
Earlier quoted context omitted.
Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
It's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.
Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.
Earlier quoted context omitted.
If payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.
Even if transport is encrypted, the LLM computations will always be clear text, right?
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE…
> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from…
Earlier quoted context omitted.
Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
It's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
Is it a fully custom inference engine or are you reusing parts of an existing stack? (llama.CPP, vLLM, etc.)
I spent a while trying to get mesh-llm running, but none of the installable llama.cpp builds worked with my older gpu. It looks like it should be able to be used to proxy an external llama.cpp service, but I had no luck setting that up either. Seems very cool, but definitely some rough edges.