41–50 of 110 posts
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
Earlier quoted context omitted.
Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again. A few questions: 1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed. 2.) Any safeguards against malicious actors poisoning model activations?
To be honest, both are very tough problems we don't have a good answer for yet. If that is something that concerns you, look into building a private mesh with trusted peers.
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.
I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
Earlier quoted context omitted.
The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet. We use a customized Q2 quantization that preserves sensitive tensors at Q8. To reduce compute time per layer, we are developing a custom GLM DSA Metal graph. While we are not yet approaching MTP, we plan to…
Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
I'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output. Nicely done OP.