It's a good project that makes sense. I recommend adding a contractual layer as well, since it's free and makes sense. Operators could legally sign that they will not look into the inference layer. After all, the operators already have a financial relationship with this provider, so it makes sense to add a contract to it and keep operators from looking into other people's data that way, too. I wish this project a lot of success.
Darkbloom – Private inference on idle Macs
211–220 of 271 posts
Re: Darkbloom – Private inference on idle Macs
#212This is one of those ideas I think makes perfect sense, but requires so much operational change for the entire stack, that it would be very difficult to scale:
- Convincing labs to run distributed, burst-y inference
- Convincing people to run their Mac all day, hoping to make a little profit
- Convincing users to trust a distributed network of un-trusted devices
I had a similar idea, pre-AI, just for compute in general. But solving even 1 of those 3 (swap AI lab for managed-compute-type-company, eg Supabase, Vercel) is nearly impossible.
Re: Darkbloom – Private inference on idle Macs
#213I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?
Of course these numbers are ridiculous. Mac Mini (let's assume Apple releases M5 Pro) tops Int8 (let's assume it is the same as FP8, which it is not) at ~50 TFLOPs, with Draw Things, we recently developed hybrid NAX + ANE inference, which can get you ~70 TFLOPs.
A H200 gives you ~4 PFLOPs, which is ~60x at only ~40x price (assuming you can get a Mac Mini at $1000). (Not to mention, BTW, RTX PRO 6000 is ~7x price for ~40x more FLOPs).
Your M4 Mac Mini only has ~20 TFLOPs.
Re: Darkbloom – Private inference on idle Macs
#214How can one do this safely? If I create a new, non-sudo user, can I install the MDM profile only for that user? I don't understand how this all works obviously so maybe this is a very dumb question
Re: Darkbloom – Private inference on idle Macs
#215the MDM profile requirement is suspect though I get why they are doing it. but it doesn't inspire confidence to see that their profile is unsigned and still using the default micromdn scep challenge...
Re: Darkbloom – Private inference on idle Macs
#216[fix: remove hardcoded API_KEYS
](https://github.com/Layr-Labs/d-inference/pull/39/changes)
Re: Darkbloom – Private inference on idle Macs
#217I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?
The numbers are optimistically legit -- it's calculated based purely considering we have demand for all machines at all times. We don't have that right now, but fairly optimistic that people will do it. That's why we don't recommend purchasing a new machine. Existing machine is no cost for you to run this. Electricity is one cost, but it will get paid off from every request it receives. Electricity is only deducted w…
I don't worry about bandwidth or constant CPU use, but the one thing that will kill my mac is burning out the SSD.
The calculator gives numbers for nearly everything, but I can't obviously see how much space it needs for model storage or how many writes of temp files I should expect if I'm running flat out.
Re: Darkbloom – Private inference on idle Macs
#218I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?
No provider maintains 100% utilization of GPUs at full rate. Demand is bursty - even if this project is successful, you might expect, e.g., things to be busy during the stock market times when Claude is throwing API errors and then severely underutilized during the same times that Anthropic was offering two-for-one off peak use. And then there's a hit for overprovisioning in general. If the network is not overprovisi…
Hold my beer: https://imgur.com/a/sNAoghL
Re: Darkbloom – Private inference on idle Macs
#219I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?
Of course these numbers are ridiculous. Mac Mini (let's assume Apple releases M5 Pro) tops Int8 (let's assume it is the same as FP8, which it is not) at ~50 TFLOPs, with Draw Things, we recently developed hybrid NAX + ANE inference, which can get you ~70 TFLOPs. A H200 gives you ~4 PFLOPs, which is ~60x at only ~40x price (assuming you can get a Mac Mini at $1000). (Not to mention, BTW, RTX PRO 6000 is ~7x price for…
> Your computer only has ~20 TFLOPs.
What a time to be alive.
Re: Darkbloom – Private inference on idle Macs
#220[fix: remove hardcoded API_KEYS ]( https://github.com/Layr-Labs/d-inference/pull/39/changes )
oh boooy, it's a benchmarking script, but still...