Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Darkbloom – Private inference on idle Macs

darkbloom.dev

Post reply on HN
Print view View original

91–100 of 271 posts

Re: Darkbloom – Private inference on idle Macs

#91

Post by subroutine »

tgma wrote: post #88
splittydev wrote: post #87

Earlier quoted context omitted.

I kind of see your point, but I also kind of don't. Sure, it would be great if you'd immediately get hammered with hundreds of requests and start make money quickly. It would also be great if it was a bit more transparent, and you could see more stats (what counts as "idle"? Is my machine currently eligible to serve models?). But it's still very new, I'd say give it some time and let's see how it goes. If you have it…

Well I already made the Ctrl+C decision. Yours may have been different, but I suppose only one of us installed it, and that one counts.

Copy?

Re: Darkbloom – Private inference on idle Macs

#93

Post by jonplackett »

chaoz_ wrote: post #16
kennywinker wrote: post #8

I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?

Solid q. I think the part of it is that it’s really easy to attract some “mass” (capital) of users, as there are definitely quite a few of idle Macs in the world. Non-VC play (not required until you can raise on your own terms!) and clear differentiation. If you want to go full-business-evaluation, I would be more worried about someone else implementing same thing with more commission (imo 95% and first to market is…

I think the point they’re making though is that the numbers seem too good to be true.

ie. Does anyone know the payback time for a B100 used just for inference? I assume it’s more than a couple of months? Or is it just training that costs so much?

Re: Darkbloom – Private inference on idle Macs

#94

Post by subroutine »

tgma wrote: post #50

I installed this so you don't have to. It did feel a bit quirky and not super polished. Fails to download the image model. The audio/tts model fails to load. In 15 minutes of serving Gemma, I got precisely zero actual inference requests, and a bunch of health checks and two attestations. At the moment they don't have enough sustained demand to justify the earning estimates.

Has anyone tested the system from the other end... sending a prompt and getting a response?

Re: Darkbloom – Private inference on idle Macs

#95

Post by [deleted] »

ramoz wrote: post #57
znnajdla wrote: post #55

Earlier quoted context omitted.

As if you get privacy with the inference providers available today? I have more trust in a randomly selected machine on a decentralized network not being compromised than in a centralized provider like OpenAI pinky promising not to read your chats.

Inference providers don't claim private inference. However, they must uphold certain security and legal compliances. You have no guarantees over any random connected laptop connected across the world.

[deleted]

Re: Darkbloom – Private inference on idle Macs

#96

Post by utopiah »

TuringNYC wrote: post #12

I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute. Any suggestions from anyone? We currently run ollama but manually manage the pools

Seems like so much more work than "just" paying for https://huggingface.co or whichever other neocloud who already did all the setup for you and just waits for your credit card per minute/seconds/token.

Re: Darkbloom – Private inference on idle Macs

#97

Post by psychoslave »

kennywinker wrote: post #8

I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?

Because they don’t have that much initial money in their pocket, while the idle computer is already there, and the biggest friction point is convincing people to install some software. Both producing rhetoric and software are several order of magnitude cheaper than to directly own and maintain a large fleet of hardware with high guarantee of getting the electrical stable input in a safe place to store them.

Assuming that getting large chunk of initial investment is just a formality is out of touch with 99% of people reality out there, when it’s actually the biggest friction point in any socio-economical endeavour.

Re: Darkbloom – Private inference on idle Macs

#98

Post by zozbot234 »

TuringNYC wrote: post #12

I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute. Any suggestions from anyone? We currently run ollama but manually manage the pools

If you set CPUSchedulingPolicy=idle Nice=19 IOSchedulingClass=idle in the ollama server configuration it should run in the background with lowest priority.

Re: Darkbloom – Private inference on idle Macs

#99

Post by gleenn »

You have to install their MDM device management software on your computer. Basically that computer is theirs now. So don't plan on just handing over your laptop temporarily unless you don't mind some company completely owning your box. Still might be a validate use for people with slightly old laptops lying around, but beware trying to share this computer with your daily activities if you e.g. use a bank on a browser on this computer regularly. MDM means they can swap out your SSL certs level of computer access, please correct me if I'm wrong.

Re: Darkbloom – Private inference on idle Macs

#100

Post by nitros »

nl wrote: post #6

They use the TEE to check that the model and code is untampered with. That's a good, valid approach and should work (I've done similar things on AWS with their TEE) The key question here is how they avoid the outside computer being able to view the memory of the internal process: > An in-process inference design that embeds the in- ference engine directly in a hardened process, elimi- nating all inter-process communi…

This entire paper smells of LLM, I'm sure even the most distinguished academic would refrain from using notation to prove that the SIP status cannot change during operation.
Post reply on HN