Cool idea, though hats off to anyone who got cohere-transcribe to show up as serving the model. I could get device to show up, but kept having issues getting their server to properly serve the model though it could just be the device I tested.
Darkbloom – Private inference on idle Macs
231–240 of 271 posts
Re: Darkbloom – Private inference on idle Macs
#232As one of the only people running a Mac Studio M3 Ultra with 512 GB of RAM on the network, I can tell you at sustained 100% GPU utilization I am measuring 250 watts max (at the power outlet). My solar panels are easily producing this. The power calculation goes away once you connect a solar panel. You can get a 400 watt solar panel on Amazon for $300.
> You can get a 400 watt solar panel on Amazon for $300. Too expensive. It's probably producing 200 watts average for 8 hours a day. That's 1600 watt hours, which is about $1.60 at PG&E prices. That would take 187 days to recoup the cost of just the panel. If you include installation costs and "what PG&E steals if you wire it to the same grid" it's probably more like 4x that, which is too long. Tell me when we can ha…
You can get them used for that price, and new for $107 if you buy qty 10+. See signature solar as one example.
Installation costs and inverters not included, however.
Re: Darkbloom – Private inference on idle Macs
#233Earlier quoted context omitted.
Macs do not have an accessible hardware TEE. Macs have secure enclaves.
Good point! But they argue that: > PT_DENY_ATTACH (ptrace constant 31): Invoked at process startup before any sensitive data is loaded. Instructs the macOS kernel to permanently deny all ptracerequests against this process, including from root. This blocks lldb, dtrace, and Instruments. > Hardened Runtime: The binary is code-signed with hardened runtime options and explicitly without the com.apple.security.get-task-a…
> PT_DENY_ATTACH
All you have to do is attach to the process before it does that, and then prevent this call from going through.
Re: Darkbloom – Private inference on idle Macs
#234I installed this so you don't have to. It did feel a bit quirky and not super polished. Fails to download the image model. The audio/tts model fails to load. In 15 minutes of serving Gemma, I got precisely zero actual inference requests, and a bunch of health checks and two attestations. At the moment they don't have enough sustained demand to justify the earning estimates.
You can see in their stats view they have a lot of providers/nodes connected but practically no actual demand/consumers. They just launched and I'm sure get providers was top of their agenda, but it's essentially unusable as a provider unless they perform some serious lift to get actual paying customers.
Re: Darkbloom – Private inference on idle Macs
#235I have a hard time believing their numbers. If you can pay off a mac mini in 2-4 months, and make $1-2k profit every month after that, why wouldn’t their business model just be buying mac minis?
The numbers are optimistically legit -- it's calculated based purely considering we have demand for all machines at all times. We don't have that right now, but fairly optimistic that people will do it. That's why we don't recommend purchasing a new machine. Existing machine is no cost for you to run this. Electricity is one cost, but it will get paid off from every request it receives. Electricity is only deducted w…
> Existing machine is no cost for you to run this.
That is not at all how modern chips work. Idle chips are mostly powered down, non-idle ones are working and that causes real measurable wear and tear on the silicon. CPU, RAM, NAND all wear and tear measurably with use on current manufacturing processes.
Re: Darkbloom – Private inference on idle Macs
#236Cool idea, though hats off to anyone who got cohere-transcribe to show up as serving the model. I could get device to show up, but kept having issues getting their server to properly serve the model though it could just be the device I tested.
Yeah I think there's a dependency issue going on there. Something isn't installed that needs to be.
Re: Darkbloom – Private inference on idle Macs
#237Earlier quoted context omitted.
If you start buying minis, then you need to house, power, and cool them. So you are building a mini data center. If you are building a small data center, economies of scale will drive you to want to build larger and larger. However, this gets expensive and neighbors tend to not like data centers (for good reason). To me this seems like asymmetric warfare against hyper-scalers.
Yup. This way, the people pay for the air conditioning themselves and they probably don't even notice the extra cost.
& if they live in a cool place, they're getting a small space heater as a bonus.
Re: Darkbloom – Private inference on idle Macs
#238bittensor has something to say about this
Re: Darkbloom – Private inference on idle Macs
#239Earlier quoted context omitted.
The numbers are optimistically legit -- it's calculated based purely considering we have demand for all machines at all times. We don't have that right now, but fairly optimistic that people will do it. That's why we don't recommend purchasing a new machine. Existing machine is no cost for you to run this. Electricity is one cost, but it will get paid off from every request it receives. Electricity is only deducted w…
> Existing machine is no cost for you to run this. That is not at all how modern chips work. Idle chips are mostly powered down, non-idle ones are working and that causes real measurable wear and tear on the silicon. CPU, RAM, NAND all wear and tear measurably with use on current manufacturing processes. https://en.wikipedia.org/wiki/Electromigration
Like pitching "drive rideshares for only the cost of your time & gas"
Re: Darkbloom – Private inference on idle Macs
#240Earlier quoted context omitted.
The numbers are optimistically legit -- it's calculated based purely considering we have demand for all machines at all times. We don't have that right now, but fairly optimistic that people will do it. That's why we don't recommend purchasing a new machine. Existing machine is no cost for you to run this. Electricity is one cost, but it will get paid off from every request it receives. Electricity is only deducted w…
> Existing machine is no cost for you to run this. That is not at all how modern chips work. Idle chips are mostly powered down, non-idle ones are working and that causes real measurable wear and tear on the silicon. CPU, RAM, NAND all wear and tear measurably with use on current manufacturing processes. https://en.wikipedia.org/wiki/Electromigration
The question is, do they wear faster than they become obsolete, as in much more expensive to run than buying a new one with higher compute/watt. (and you can also factor in the ability to run latest models at usable speed)