Use One Big Server (2022) • Hacker News Board

Re: Use One Big Server (2022)

#271

Post by tgtweak » Mon, Sep 01, 2025, 4:50 PM UTC

synack wrote: post #151

The complexity you introduce trying to achieve 100% uptime will often undermine that goal. Most businesses can tolerate an hour or two of downtime or data loss occasionally. If you set this expectation early on, you can engineer a much simpler system. Simpler systems are more reliable.

We had single-datacenter resiliency (meaning n+1 on power, cooling, network + isp, servers) and it was fine. You still need offsite DRS strategy here - this is one of the things having that hybrid cloud is great for: you can replicate your critical workloads like databases and services to the cloud in no-load standby, or delta-copy your backups to a cheap cloud provider for simplified recovery in a disaster scenario (ie: entire datacenter gets taken out). The cost of this is relatively low since data into the cloud is free and you're only really incurring costs in a disaster recovery scenario. Most virtualized platforms (veeam etc) support offsite secondary incremental backups with relative ease, recovery is also pretty straightforward.

That being said I've lost a lot of VMs on ec2 and had entire regions go down in gcp and aws in the last 3 years alone, so going to the public cloud isn't a solves it all solution - knock on wood the colo we've been using hasn't been down once in 12+ years.

Re: Use One Big Server (2022)

#272

Post by amluto » Mon, Sep 01, 2025, 4:53 PM UTC

rthnbgrredf wrote: post #264

Bare-metal servers sound super cheap when you look at the price tag, and yeah, you get a lot of raw power for the money. But once you’re in an enterprise setup, the real cost isn’t the hardware at all, it’s the people needed to keep everything running. If you go this route, you’ve got to build out your own stack for security, global delivery, databases, storage, orchestration, networking ... the whole deal. That mean…

If you use a cloud, you need a solution for security (ever heard of “shared responsibility”?), global delivery (a big cloud will host you all over, and this requires extra effort on your part, kind of like how having multiple rented or owned servers requires extra effort), storage (okay, I admit that S3 et al are nice and that non-big-cloud solutions are a bit lacking in this department), orchestration (the cloud handles only the lowest level — you still need to orchestrate your stuff on top of it), fixing breakage at 3 a.m. (the cloud can swap you onto a new server, subject to availability; so can a provider like Hetzner. You still need to fail over to that server successfully), patching stuff (other than firmware, the cloud does not help you here).

Re: Use One Big Server (2022)

#273

Post by radiator » Mon, Sep 01, 2025, 4:56 PM UTC

benjiro wrote: post #211

runako wrote: post #26

One of the more detrimental aspects of the Cloud Tax is that it constrains the types of solutions engineers even consider. Picking an arbitrary price point of $200/mo, you can get 4(!) vCPUs and 16GB of RAM at AWS. Architectures are different etc., but this is roughly a mid-spec dev laptop of 5 or so years ago. At Hetzner, you can rent a machine with 48 cores and 128GB of RAM for the same money. It's hard to overstat…

> At Hetzner, you can rent a machine with 48 cores and 128GB of RAM for the same money. The problem that Hetzner and a lot of hardware providing hosts have, is the lack of affordable flexibility. Hetzner their design is based upon a base range of standardized products. This can only be upgraded within a pre-approved range of upgrade options (limited to storage/memory). Upgrades are often a mixed bag of carefully desi…

> Doubling the storage on a base server, often increases the price of your server by 50 to 75%

For storage, Hetzner does offer Volumes, which you can attach to your VM and you can choose exactly how large you want them to be and are charged separately. But your argument about doubling resources and doubling prices still holds for RAM.

Re: Use One Big Server (2022)

#274

Post by andersmurphy » Mon, Sep 01, 2025, 4:56 PM UTC

immibis wrote: post #257

wild_egg wrote: post #255

Earlier quoted context omitted.

You know, it's ok to say that you're out of your element and don't have direct experience with the thing you're commenting on. SQLite is easily the best scaling DB tech I've used. I've moved all my postgres workloads over to it and the gains have been incredible. It's not a panacea and not the best in all cases but it's a very sane default that I recommend everyone start with and only complicate their stack with an e…

> You know, it's ok to say that you're out of your element and don't have direct experience with the thing you're commenting on. I moved several projects from sqlite to postgres because sqlite didn't scale enough for any of them.

May I suggest you could have been holding it wrong?

The out of the box defaults for sqlite are terrible for web apps.

Re: Use One Big Server (2022)

#275

Post by msgodel » Mon, Sep 01, 2025, 4:57 PM UTC

rthnbgrredf wrote: post #264

Bare-metal servers sound super cheap when you look at the price tag, and yeah, you get a lot of raw power for the money. But once you’re in an enterprise setup, the real cost isn’t the hardware at all, it’s the people needed to keep everything running. If you go this route, you’ve got to build out your own stack for security, global delivery, databases, storage, orchestration, networking ... the whole deal. That mean…

I used to say "oh yeah just run qemu-kvm" until my girlfriend moved in with me and I realized you do legitimately need some kind of infrastructure for managing your "internal cloud" if anyone involved isn't 100% on the same page and then that starts to be its own thing you really do have to manage.

Suddenly I learned why my employer was willing to spend so much on OpenStack and Active directory.

Re: Use One Big Server (2022)

#276

Post by rollcat » Mon, Sep 01, 2025, 5:06 PM UTC

rthnbgrredf wrote: post #264

Bare-metal servers sound super cheap when you look at the price tag, and yeah, you get a lot of raw power for the money. But once you’re in an enterprise setup, the real cost isn’t the hardware at all, it’s the people needed to keep everything running. If you go this route, you’ve got to build out your own stack for security, global delivery, databases, storage, orchestration, networking ... the whole deal. That mean…

A single, powerful box (or a couple, for redundancy) may still be the right choice, depending on your product / service. Renting is arguably the most approachable option: you're outsourcing the most tedious parts + you can upgrade to a newer generation whenever it becomes operationally viable. You can add bucket storage or CDN without dramatically altering your architecture.

Early Google rejected big iron and built fault tolerance on top of commodity hardware. WhatsApp used to run their global operation employing only 50 engineering staff. Facebook ran on Apache+PHP (they even served index.php as plain text on one occasion). You can build enormous value through simple means.

Re: Use One Big Server (2022)

#277

Post by dang » Mon, Sep 01, 2025, 5:09 PM UTC

d_burfoot wrote: post #243

dang wrote: post #48

HN uses two—one live and one backup, so we can fail over if there's a hardware issue or we need to upgrade something. It's a nice pattern. Just don't make them clones of each other, or they might go BLAM at the same time! https://news.ycombinator.com/item?id=32049205 https://news.ycombinator.com/item?id=32032235 https://news.ycombinator.com/item?id=32028511 ( --- Edit: both these points are mentioned in the OP.

Any stats on HN downtime over the years? I remember one or two outages in the last decade or so, but I would guess the uptime is about 99.99%.

We don't specifically track that, no. The worst one was when we went down for (IIRC) a couple days because of a disk failure, I think in Jan 2014. It was after that that we added a failover box.

HN goes down when we restart the server process, usually as part of updating the code - but only for a few seconds. The message "Restarting the server. Shouldn't take long." displays when that is happening.

There are also, to my exasperation, still moments of brownout during certain traffic spikes or moments of obscure resource contention. But these are at least rarer than they used to be.

Re: Use One Big Server (2022)

#278

Post by pdhborges » Mon, Sep 01, 2025, 5:32 PM UTC

benterix wrote: post #201

bspammer wrote: post #84

Earlier quoted context omitted.

A fair comparison would include the cost of the DBA who will be responsible for backups, updates, monitoring, security and access control. That’s what RDS is actually competing with.

You are aware that RDS needs backups, setting up monitoring properly, defining access, providing secrets management etc., and updates between major versions are not automatic? RDS has a value. But for many teams the price paid for this value is ridiculously high when compared to other options.

AWS can make major version upgrades automatically now with less downtime. I think they do the logical replication dance internally.

Re: Use One Big Server (2022)

#279

Post by ahdanggit » Mon, Sep 01, 2025, 5:43 PM UTC

msgodel wrote: post #275

rthnbgrredf wrote: post #264

Bare-metal servers sound super cheap when you look at the price tag, and yeah, you get a lot of raw power for the money. But once you’re in an enterprise setup, the real cost isn’t the hardware at all, it’s the people needed to keep everything running. If you go this route, you’ve got to build out your own stack for security, global delivery, databases, storage, orchestration, networking ... the whole deal. That mean…

I used to say "oh yeah just run qemu-kvm" until my girlfriend moved in with me and I realized you do legitimately need some kind of infrastructure for managing your "internal cloud" if anyone involved isn't 100% on the same page and then that starts to be its own thing you really do have to manage. Suddenly I learned why my employer was willing to spend so much on OpenStack and Active directory.

> until my girlfriend moved in with me

lol, why was this the defining moment? She wasn't too keen on hearing the high pitch wwwwhhhhuuuuurrrrrrr of the server fans?

Re: Use One Big Server (2022)

#280

Post by ahdanggit » Mon, Sep 01, 2025, 5:46 PM UTC

I used a colo once a few years ago at a small datacenter in the midwest, I was shocked at how unprofessional everything was, machines laying in the hallway, a guy was sleeping in one of the offices. They let me setup my server and was left unattended several times, I could have just poked the power button on a nearby server or moved a cable or whatever. It was a 1.5 hour drive away, and I wasn't running anything serious so I just went with it but pulled my stuff out after my 1 year subscription was up.