Why isn’t a MacBook Air M5 on the hardware list?
Because the model that generated that list was trained before the M5 came out.
Darkbloom – Private inference on idle Macs
141–150 of 271 posts
Re: Darkbloom – Private inference on idle Macs
#142I installed this so you don't have to. It did feel a bit quirky and not super polished. Fails to download the image model. The audio/tts model fails to load. In 15 minutes of serving Gemma, I got precisely zero actual inference requests, and a bunch of health checks and two attestations. At the moment they don't have enough sustained demand to justify the earning estimates.
weird to learn that they do not generate inference requests to their network themselves to motivate early adopters at least to host their inference software
If they paid promised > $1k/m for FLUX 2B on a Mac they would go broke in less than a month. On a single 5090 that model would provide an inference througput so high they'd have to pay close to $50k/m for the results.
The numbers are absolute fraud. You shouldn't be installing their software cause fraud could be not just about numbers.
Re: Darkbloom – Private inference on idle Macs
#143They use the TEE to check that the model and code is untampered with. That's a good, valid approach and should work (I've done similar things on AWS with their TEE) The key question here is how they avoid the outside computer being able to view the memory of the internal process: > An in-process inference design that embeds the in- ference engine directly in a hardened process, elimi- nating all inter-process communi…
While they do make this argument, realistically anyone sending their prompt/data to an external server should assume there will be some level of retention. And more so in particular, anyone using Darkbloom with commercial intents should only really send non-sensitive data (no tokens, customer data, ...) I'd say only classification tasks, imagine generation, etc.
There's a difference between trusting Anthropic and trusting random mac owners.
Re: Darkbloom – Private inference on idle Macs
#144Earlier quoted context omitted.
and I don't think they ever will unless they're highly competitive (hopefully that price they have stays? at least for users) I was thinking of building this exact thing a year ago but my main stopper was economics: it would never make sense for someone to use the API, thus nobody can make money off of zero demand. I guess we just have to look at how Uber and Airbnb bootstrapped themselves. Another issue with my orig…
What's a good place to do this?
For Windows there's https://borg.games/setup (I'm the author).
Re: Darkbloom – Private inference on idle Macs
#145> Every request is end-to-end encrypted Afaik you will need to decrypt the data the moment it needs to be fed into the model. How do they do this then?
The system hosting the model must be one of the ends.
Remember, all encryption is E2EE if you're not picky about the ends.
Re: Darkbloom – Private inference on idle Macs
#146Cool idea. Just some back-of-the-envelope math here (not trusting what's on their site): My M5 Pro can generate 130 tok/s (4 streams) on Gemma 4 26B. Darkbloom's pricing is $0.20 per Mtok output. That's about $2.24/day or $67/mo revenue if it's fully utilized 24/7. Now assuming 50W sustained load, that's about 36 kWh/mo, at ~$.25/kWh approx. $9/mo in costs. Could be good for lunch money every once in a while! Around…
Their example big earner models are FLUX.2 Klein 4B and FLUX.2 Klein 9B, which i imagine could generate a lot more tokens/s than a 26B model on your machine. For Gemma 4 26B their math is: single_tok/s = (307 GB/s / 4 GB) * 0.60 = 46.0 tok/s batched_tok/s = 46.0 * 10 * 0.9 = 414.4 tok/s tok/hr = 414.4 * 3600 = 1,492,020 revenue/hr = (1,492,020 / 1M) * $0.200000 = $0.2984 I have no idea if that is a good estimate of h…
Interesting token numbers they're using, because I've benchmarked it at 69 tok/s single steam and 130 multi stream.
Re: Darkbloom – Private inference on idle Macs
#147Cool idea. Just some back-of-the-envelope math here (not trusting what's on their site): My M5 Pro can generate 130 tok/s (4 streams) on Gemma 4 26B. Darkbloom's pricing is $0.20 per Mtok output. That's about $2.24/day or $67/mo revenue if it's fully utilized 24/7. Now assuming 50W sustained load, that's about 36 kWh/mo, at ~$.25/kWh approx. $9/mo in costs. Could be good for lunch money every once in a while! Around…
> My M5 Pro can generate 130 tok/s (4 streams) on Gemma 4 26B. This seems high. At which quantization? Using LM Studio or something else? Note: Darkbloom seems to run everything on Q8 MLX.
Ah good point, this is using Q4, benchmarked total throughout serving with Llama.cpp.
Re: Darkbloom – Private inference on idle Macs
#148Earlier quoted context omitted.
No it's not the same premise at all. Crypto doesn't do anything useful for legitimate businesses. AI inference is very useful for legitimate businesses, and so are residential IP proxies for scraping. And by definition, residential IPs cannot be centralized. And as building GPUs becomes more expensive, the existing pool of second hand unused hardware becomes more valuable, not less. The problem with crypto mining is…
It is the same premise, because the person you are responding to is not talking about the moral implications at all, only about the financial / hardware implications. Running AI inference increases the power draw, and requires certain hardware. Mining bitcoin increases the power draw, and requires certain hardware. OP's point thus stands: Bad players will find places to get far cheaper power than the intended audienc…
The main problem with crypto is there is no universal need for it. The demand for crypto doesn’t keep increasing as compute gets cheaper. But the demand for AI inference is only growing, and making it cheaper would likely only increase demand. So it’s not a race to the bottom. Sure hyper focused players can earn more at higher margins. But average players can probably still earn decently. Take for example electricity. It can still be profitable for a home in Germany to install balcony solar and make a little money selling back to the grid even though it’s obviously not as efficient as an industrial power plant. Mom and pop AI inference don’t have to be super efficient as long as they serve a universal need - it will be like balcony solar in Europe.
Re: Darkbloom – Private inference on idle Macs
#149Thanks, if this takse off. I have finally some motivation to do exploitation in kernel. :)
Re: Darkbloom – Private inference on idle Macs
#150hey guys! i'm the creator. let me know if you have any questions.