301–310 of 516 posts
Old hardware is _plenty_ powerful for a lot of tasks today.
Earlier quoted context omitted.
From what I've seen, if you're depending on AWS, if something fails you too need someone 24x7 so that you can take action as well. Sometimes magic happens and systems recover after aws restarts their DNS, but usually the combination of event causes the application to get into an unrecoverable state that you need manual action. It doesn't always happen but you need someone to be there if it ever happens. Or bare minim…
How many problems is AWS able to handle for you that you are never aware of though?
I've only had one outage I could attribute to running on-prem, meanwhile it's a bit of a joke with the non-IT staff in the office that when "The Internet" (i.e. Cloudflare, Amazon) goes down with news reports etc our own services are all running fine.
Earlier quoted context omitted.
Capex may also require you to take out loans
Which is incredibly difficult in the public sector. Yes, there are various financing instruments available for capital purchases but they're always annoying, slow and complicated. It's much easier to spend 5k per month than 500k outright.
I think the primary reason that people over fixate on the cloud is that they can't do math. So renting is a hedge.
The argument made 2 decades ago was that you shouldn't own the infrastructure (capital expense) and instead just account for the cost as operational expense (opex). The rationale was you exchange ownership for rent. Make your headache someone else's headache.
The ping pong between centralized vs decentralized, owned vs rented, will just keep going. It's never an either or, but when companies make it all-or-nothing then you have to really examine the specifics.
Looks insanely daunting imo
> Self-reliance is great, but there are other benefits to running your own compute. It inspires good engineering. It's easy to inspire people when you have great engineers in the first place. That's a given at a place like comma.ai, but there are many companies out there where administering a datacenter is far beyond their core competencies. I feel like skilled engineers have a hard time understanding the trade-offs…
They spend too much time on yet another cloud native support group call, learning for ThatOneCloudProvider certificates, figuring out that single implementation caveats, standardizing security procedures between cloud teams, and so on.
Yet people in the article just throw a 1000 lines of code KV store mkv [0] on a huge raw storage server and call it a day. And it's a legit choice, they did actual study beforehand and concluded: we don't need redundancy in most cases. At all. I respect that.
Feels like I’ve lived through a full infrastructure fashion cycle already. I started my career when cloud was the obvious answer and on-prem was “legacy.” Now on-prem is cool again. Makes me wonder whether we’re already setting up the next cycle 10 years from now, when everyone rediscovers why cloud was attractive in the first place and starts saying “on-prem is a bad idea” again.
My entire career I’ve encountered people passionately pushing for on-prem and railing against anything cloud. I can’t remember a time when Hacker News comments leaned pro-cloud because it’s always been about self-hosting.
The few times the on-prem people won out in my career never went exactly as they imagined. Buying a couple servers and setting them up at the colo is easy enough, but the slow and steady drag of maintaining your own infrastructure starts to work its way into every development cycle after that. In my experience, every team has significantly underestimated how all the little things add up to a drag on available time for other work.
The best case for on-prem that I saw was when a company was basically in maintenance mode. Engineers had a lot of extra time to optimize, update. maintain, and cost reduce without subtracting from feature development or bug fixes.
The worst cases for on-prem I’ve seen have been funded startups. In this situation it’s imperative that everyone focus on feature development and rapid iteration. Letting some of the engineers get sidetracked with setting up and maintaining their own hosting to save a dollar amount that barely hires 1-2 more engineers but sets the schedule back by many months was a huge mistake.
In my experience, most engineers become less enchanted with rolling their own on premises hosting as they get older. Their work becomes more about getting the job done quickly and to budget, not hyper-optimizing the hosting situation at the expense of inviting more complexity and miscellaneous tasks into their workload.
Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…
Earlier quoted context omitted.
It's worth becoming good at.
Is it though? This is a genuine question. My intuition is that the investment of time / stress / risk to become good at this is unlikely to have high ROI to either the person putting in that time or to the business paying them to do so. But maybe that's not right.
Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery.
We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of.
Add monitoring that the DB servers are working, replication is up-to-date and the backups are working.
Sure you have to schedule your own hardware repairs or updates but it also means you don't need to wrangle with the ridiculous cost-engineering, reserved instances, cloud product support issues or API deprecations, proprietary configuration languages, etc.
Bare metal is better for a lot of non-cost reasons too, as the article notes it's just easier/better to reason about the lower level primitives and you get more reliable and repeatable performance.