Darkbloom – Private inference on idle Macs
darkbloom.dev
Darkbloom – Private inference on idle Macs
1–10 of 271 posts
Re: Darkbloom – Private inference on idle Macs
#2Why only Macs? If we think of all PCs and mobile phones running idle, the potential is much larger.
Re: Darkbloom – Private inference on idle Macs
#3Why only Macs? If we think of all PCs and mobile phones running idle, the potential is much larger.
simple first target, PCs have more variability
Re: Darkbloom – Private inference on idle Macs
#4Should have called it “Inferanet” with this idea.
Away this looks like a great idea and might have a chance at solving the economic issue with running nodes for cheap inference and getting paid for it.
Re: Darkbloom – Private inference on idle Macs
#5Why only Macs? If we think of all PCs and mobile phones running idle, the potential is much larger.
From the paper: https://github.com/Layr-Labs/d-inference/blob/master/papers/...
> Apple’s attestation servers will only generate the FreshnessCode for a genuine device that checks in via APNs. A software-only adversary cannot forge the MDA certificate chain (Assumption 3). Com- bined with SIP enforcement (preventing binary replace- ment) and Secure Boot (preventing bootloader tampering), this provides strong evidence that the signing key resides in genuine Apple hardware.
Re: Darkbloom – Private inference on idle Macs
#6They use the TEE to check that the model and code is untampered with. That's a good, valid approach and should work (I've done similar things on AWS with their TEE)
The key question here is how they avoid the outside computer being able to view the memory of the internal process:
> An in-process inference design that embeds the in- ference engine directly in a hardened process, elimi- nating all inter-process communication channels that could be observed, with optional hypervisor mem- ory isolation that extends protection from software- enforced to hardware-enforced via ARM Stage 2 page tables at zero performance cost.[1]
I was under the impression this wasn't possible if you are using the GPU. I could be misled on this though.
[1] https://github.com/Layr-Labs/d-inference/blob/master/papers/...
Re: Darkbloom – Private inference on idle Macs
#7Why only Macs? If we think of all PCs and mobile phones running idle, the potential is much larger.
They use the Apple TEE which they claim also protects GPU memory (I wasn't aware of this).
NVidia data center GPUs have a similar path, but not their consumer ones. Not sure about the NVidia Spark.
It's possible AMD Strix Halo can do this, but unlikely for any other PC based GPU environments.
Re: Darkbloom – Private inference on idle Macs
#8I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?
Re: Darkbloom – Private inference on idle Macs
#9That solution actually makes great sense. So Apple won in some strange way again?
Guess there are limitations on size of the models, but if top-tier models will getting democratized I don’t see a reason not to use this API. The only thing that comes to me is data privacy concerns.
I think batch-evals for non-sensitive data has great PMF here.
Re: Darkbloom – Private inference on idle Macs
#10I thought this was Apple’s plan all along. How is this not already their thing?