Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hold on to Your Hardware

xn--gckvb8fzb.com

Post reply on HN
Print view View original

231–240 of 559 posts

Re: Hold on to Your Hardware

#231

Post by mwcz »

saadn92 wrote: post #201

The article's dystopia section is dramatic but the practical point is real. I've been self-hosting more and more over the past year specifically because I got uncomfortable with how much of my stack depended on someone else's servers. Running a VPS with Tailscale for private access, SQLite instead of managed databases, flat files synced with git instead of cloud storage. None of this requires expensive hardware, it j…

Depending on someone else's servers isn't that different from depending on someone else's software, which unfortunately we all must do. Unfathomable reams of it, with a growth curve that recently went vertical. I guess the crucial difference is that someone else's servers can be taken away in a flash, while someone else's (FOSSl software can't.

Re: Hold on to Your Hardware

#232

Post by TeMPOraL »

3form wrote: post #158
TeMPOraL wrote: post #115

Earlier quoted context omitted.

If only there wasn't a security update treadmill forcing everyone to do regular hardware upgrades.

Of course, as long as we're in the dreamland, most of these security upgrades do not actually require a hardware upgrade.

Technically no (except for the gradual performance drop they introduce, + occasional TPM bullshit), but of course in practice, companies see this as a choice of spending money on back-porting security fixes to a growing range of hardware, vs. making money by not doing that and forcing everyone to buy new hardware instead.

Re: Hold on to Your Hardware

#233

Post by nhecker »

zozbot234 wrote: post #209
nhecker wrote: post #153

Earlier quoted context omitted.

In a similar vein: seek efficiency. I.e., /if/ I am going to consume LLM tokens, I figure that a local LLM with 10s of billions of parameters running on commodity hardware at home will still consume far more energy per token than that of a frontier model running on commercial hardware which is very strongly incentivized to be as efficient as possible. Do the math; it isn't even close. (Maybe it'd be closer in your lo…

But commodity hardware that's right-sized for your own private needs is many orders of magnitude cheaper than datacenter hardware that's intended to serve millions of users simultaneously while consuming gigawatts in power. You're mostly paying for that hardware when you buy LLM tokens, not just for power efficiency. And your own hardware stays available for non-AI related needs, while paying for these tokens would r…

>And your own hardware stays available for non-AI related needs, while paying for these tokens would require you to address these needs separately in some way.

^ Fair. Yep, I agree the calculus changes if you don't have _any_ local hardware and you're needing to factor in the cost of acquiring such hardware.

When I did this napkin math, I was mostly interested in the energy aspect, using cost as a proxy. I was calculating the $/token (taking into consideration the cost of a KWh from my utility, the measured power draw of my M1 work machine, and the measured tokens per second processed by a ~20BP open-weight model). I then compared this to the published $/token rate of a frontier provider, and it was something like two orders of magnitude in favor of the frontier model. I get it, they're subsidizing, but I've got to imagine there's some truth in the numbers.

I wonder, does (or will) the $/token ratio fall asymptotically toward the cost of electricity? In my mind I'm drawing a parallel to how the value of mined cryptocurrency approximately tracks the cost of electricity... but I might be misremembering that detail.

Re: Hold on to Your Hardware

#234

Post by lenova »

Oh man, I've come across this person's blog before and I love it, not just because of the personalization/personality they've put into the site's design, but because of all of the random CLI/TUI-based tools they've developed. Examples:

- https://xn--gckvb8fzb.com/projects/

Their github repos:

- https://github.com/mrusme

They even built a BBS-style reader client that supports Hacker News:

https://github.com/mrusme/neonmodem

I miss the days of the web being weird like this :-)

Re: Hold on to Your Hardware

#235

Post by mbeex »

gregoriol wrote: post #212
saadn92 wrote: post #201

The article's dystopia section is dramatic but the practical point is real. I've been self-hosting more and more over the past year specifically because I got uncomfortable with how much of my stack depended on someone else's servers. Running a VPS with Tailscale for private access, SQLite instead of managed databases, flat files synced with git instead of cloud storage. None of this requires expensive hardware, it j…

You are missing one important part: maintenance. While on a managed service, dozens of hours of maintenance are done by someone, when you are self-hosting, you'll be doing 3 times that, because you can't know all the details of making so many tools work, because each tool will have to be upgraded at some point and the upgrade will fail, because you have to test you backups, and many many more things to do in the long…

Much easier with AI. Went from Webhosting all-in package + NAS to Hetzner Storage Share and a separate Emailprovider (Runbox). After a short time I dumped the Nextcloud instance and moved on to a Hetzner VPS with five docker containers, Caddy, proper authentication and all. Plus a Storage Box. Blogging/Homepage as Cloudflare Pages, fed by Github, domains from CF and porkbun, Tailscale, etc., etc. ad nauseam, NAS still there.

Most of this I didn't for many years because it is not my core competence (in particular the security aspects). Properly fleshed-out explanations from any decent AI will catapult you to this point in no time. Maintenance? Almost zero.

p.s. Admittedly, it's not a true self-hosting solution, but the approach is similar and ultimately leads to that as well.

Re: Hold on to Your Hardware

#236

Post by TeMPOraL »

adrianN wrote: post #116
TeMPOraL wrote: post #64

Earlier quoted context omitted.

Well, as long as they can make electricity too cheap to meter, we can get helium from somewhere . Mine it from LNG sources currently untapped due to EROI < 1, or ship it from the goddamn Moon - ultimately, every problem in life (except that of human heart) can be solved with cheap energy.

There is to my knowledge no reason to assume that complicated physics experiments that heat water to run a steam engine will be much cheaper than fission power plants, unfortunately.

I can't say I agree with the conclusion, but I commend you for the concise and poetic description of what most power plants fundamentally are.

Re: Hold on to Your Hardware

#237

Post by embedding-shape »

Aurornis wrote: post #215
barrkel wrote: post #15

I don't buy the central thesis of the article. We won't be in a supply crunch forever. However, I do believe that we're at an inflection point where DC hardware is diverging rapidly from consumer compute. Most consumers are using laptops and laptops are not keeping pace with where the frontier is in a singular compute node. Laptops are increasingly just clients for someone else's compute that you rent, or buy a time…

> Most consumers are using laptops and laptops are not keeping pace with where the frontier is in a singular compute node. How can you say this when Apple is releasing extremely fast M5 MacBook Pros? Or the $600 MacBook Neo that has incredible performance for that price point? Even x86 is getting some interesting options. The Strix Halo platform has become popular with LLM users that the parts are being sold in high…

They're ultimately laptops, you won't be able to squeeze out the same amount of performance from a laptop compared to a desktop, regardless of the hardware.

If you haven't tried out a desktop CPU in a while, I highly recommend you giving it a try if you're used to only using laptops, even when in the same class the difference is obvious.

Re: Hold on to Your Hardware

#238

Post by duskdozer »

darkwater wrote: post #120

> For the better part of two decades, consumers lived in a golden age of tech. Memory got cheaper, storage increased in capacity and hardware got faster and absurdly affordable. I got my first PC circa 1992 (a 2nd hand IBM PS/2, 80286 processor with 2MB RAM and 30MB HDD) and the "golden age" was already there. We are well over 40 years of almost uninterrupted "pay less for more performances" in the home/personal comp…

Do you mean for hardware? Because a big chunk of that imo is how unnecessarily demanding software has become in the last 10 years, largely due to the web.

Re: Hold on to Your Hardware

#240

Post by nhecker »

audunw wrote: post #188
TeMPOraL wrote: post #102

Earlier quoted context omitted.

Not necessarily. Many people grew up with PCs and laptops but now mostly use their phones, because outside of specific jobs or hobbies, everyday computing needs are heavily optimized for mobile-first. (A large factor here is, obviously, the cloud. With photos, documents, e-mail, IMs, etc. all hosted for cheap or free on "other people's computers", the total hardware demands on the end-user computing device is much le…

> Not necessarily. Many people grew up with PCs and laptops but now mostly use their phones, because outside of specific jobs or hobbies, everyday computing needs are heavily optimized for mobile-first. It's a deeply flawed comparison, because many of the things we do with a phone now wasn't something we'd do at all with the computers we grew up with. We didn't pay at the grocery store with a computer, we didn't buy…

(edit: I'm broadly in agreement with your comment & observations, so I don't at all mean to come off as argumentative for the sake of being argumentative. You just got me thinking about how that situation might have been handled thirty or a hundred years ago.)

> [...] my doctor can now approve my request for a prescription from anywhere in the world. That just wasn't possible before [...]

I'm picking nits, but wasn't this more or less instantaneous approval possible before with e.g., a fax and a telephone? Or (although this is a bit of a stretch) a telegram and telegraph?

Post reply on HN