Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Darkbloom – Private inference on idle Macs

darkbloom.dev

Post reply on HN
Print view View original

131–140 of 271 posts

Re: Darkbloom – Private inference on idle Macs

#131

Post by torginus »

pants2 wrote: post #17

Cool idea. Just some back-of-the-envelope math here (not trusting what's on their site): My M5 Pro can generate 130 tok/s (4 streams) on Gemma 4 26B. Darkbloom's pricing is $0.20 per Mtok output. That's about $2.24/day or $67/mo revenue if it's fully utilized 24/7. Now assuming 50W sustained load, that's about 36 kWh/mo, at ~$.25/kWh approx. $9/mo in costs. Could be good for lunch money every once in a while! Around…

Also this assumes hardware never fails. I learned about this the hard way back when I started mining crypto on my 5700XT way back when.

I figured since I already used it a lot, and I've never had a GPU fail on me, it would be fine.

The fans on it died in a month of constant use, replacing them was more money than what I made on mining.

Re: Darkbloom – Private inference on idle Macs

#132

Post by znnajdla »

aacid wrote: post #128
znnajdla wrote: post #53

Earlier quoted context omitted.

The numbers are obviously high, because if this takes off then the price for inference will also drop. But I still think it’s a solid economic model that benefits low income countries the most. In Ukraine, for example, I know people who live on $200/month. A couple Mac Minis could feed a family in many places. As a business owner, I can think of multiple reasons why a decentralized network is better for me as a busin…

Isn't this same premise as "lets buy few GPUs to mine crypto and have passive income"? It didn't work very well and it probably won't work now either. If there is money to be made, bigger players will get in there, buy out all mac minis they can, drive price up for regular people and inevitably drive inference price down so you'll be lucky to get initial investment back

No it's not the same premise at all. Crypto doesn't do anything useful for legitimate businesses. AI inference is very useful for legitimate businesses, and so are residential IP proxies for scraping. And by definition, residential IPs cannot be centralized. And as building GPUs becomes more expensive, the existing pool of second hand unused hardware becomes more valuable, not less. The problem with crypto mining is that it quickly becomes unprofitable for small scale deployments. I'm not sure if AI inference would be, especially for the decentralized benefits of lower latency.

Re: Darkbloom – Private inference on idle Macs

#134

Post by Inferlane »

NiloCK wrote: post #110

Interesting to see an offering with this heritage [1] proposing flat earnings rates for inference operators here, rather than trying to sell a dynamic marketplace where operators compete on price in real-time. Right now the dashboards show 78 providers online, but someone in-thread here said that they spun one up and got no requests. Surely someone would be willing to beat the posted rate and swallow up the demand? I…

[dead]

Re: Darkbloom – Private inference on idle Macs

#135

Post by usrusr »

splittydev wrote: post #87
tgma wrote: post #80

Earlier quoted context omitted.

You'd think to bootstrap a marketplace you'd spend your own money to feed fake requests (or perhaps allow free chat so that they induce requests). Still, absolute zero is an unacceptable number. Had this running for more than an hour.

I kind of see your point, but I also kind of don't. Sure, it would be great if you'd immediately get hammered with hundreds of requests and start make money quickly. It would also be great if it was a bit more transparent, and you could see more stats (what counts as "idle"? Is my machine currently eligible to serve models?). But it's still very new, I'd say give it some time and let's see how it goes. If you have it…

Bootstrapping will be near-impossible (or incredibly costly) unless they offer inference consumers models with established demand arriving at some least-cost router service where they can undercut the competition (if they actually can). And then dogfood the opportunistic provider side on their own Macs, but with a preference to putting third parties first in the queue. Everything else is just wishful thinking.

Re: Darkbloom – Private inference on idle Macs

#136

Post by heddycrow »

I think it’s important that systems like this exist, but getting them off the ground is non-trivial.

We’ve been building something similar for image/video models for the past few months, and it’s made me think distribution might be the real bottleneck.

It’s proving difficult to get enough early usage to reach the point where the system becomes more interesting on its own.

Curious how others have approached that bootstrap problem. Thanks in advance.

Re: Darkbloom – Private inference on idle Macs

#137

Post by rzwitserloot »

znnajdla wrote: post #132
aacid wrote: post #128

Earlier quoted context omitted.

Isn't this same premise as "lets buy few GPUs to mine crypto and have passive income"? It didn't work very well and it probably won't work now either. If there is money to be made, bigger players will get in there, buy out all mac minis they can, drive price up for regular people and inevitably drive inference price down so you'll be lucky to get initial investment back

No it's not the same premise at all. Crypto doesn't do anything useful for legitimate businesses. AI inference is very useful for legitimate businesses, and so are residential IP proxies for scraping. And by definition, residential IPs cannot be centralized. And as building GPUs becomes more expensive, the existing pool of second hand unused hardware becomes more valuable, not less. The problem with crypto mining is…

It is the same premise, because the person you are responding to is not talking about the moral implications at all, only about the financial / hardware implications.

Running AI inference increases the power draw, and requires certain hardware.

Mining bitcoin increases the power draw, and requires certain hardware.

OP's point thus stands: Bad players will find places to get far cheaper power than the intended audience, and will buy dedicated hardware, at which point the money you can earn to do this will soon drop below the costs for power (for folks like you and me).

Maybe that won't happen, but why won't that happen?

Re: Darkbloom – Private inference on idle Macs

#138

Post by haspok »

Having strong SETI@Home vibes from 25 years ago, except of course, this is not for the greater good of humanity, but a for-profit project.

Problem is, from a technical point of view, what kind of made sense back then (most people running desktops, fans always on, energy saving minimal) is kind of stupid today (even if your laptop has no fan, would you want it to be always generating heat?)...

I definitely want my laptops to be cool, quiet and idle most of the time.

Re: Darkbloom – Private inference on idle Macs

#140

Post by vorticalbox »

haspok wrote: post #138

Having strong SETI@Home vibes from 25 years ago, except of course, this is not for the greater good of humanity, but a for-profit project. Problem is, from a technical point of view, what kind of made sense back then (most people running desktops, fans always on, energy saving minimal) is kind of stupid today (even if your laptop has no fan, would you want it to be always generating heat?)... I definitely want my lap…

I some times play about with local models via ollama/comfyui and more recently ace-step to generate music.

This is short bursts of heat 5-10 m during the render I would not be happy with that for multiple hours a day. I am sure that would have a negative effect on battery health.

Post reply on HN