Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS down again?

aws.amazon.com

Post reply on HN
Print view View original

161–170 of 257 posts

Re: AWS down again?

#161

Post by Spivak »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

(Assumedly) much much younger person here who has spent the majority of her career on prem, has lifted multiple shops to AWS, and right now works full time on an all AWS stack.

Your experience with UNIX isn't worthless, but it's worthless to anyone who's working "further up the stack" than you. If you feel like your skills are degrading then you need to find a job somewhere that's actually building infra that shops will build on top of. Your skills aren't day-to-day anymore and that's a good thing. Your generation made all that junk turnkey in the way that you probably think about dealing with Ethernet frames. Taking my first AWS job literally obsoleted all my system's knowledge -- super humbling experience -- it wasn't completely worthless but none of the problems I've spent years working out solutions even existed anymore.

You're confusing people building products with people building infrastructure -- the devops role makes this messy because you're usually "using" infra tools like a dev rather than building them. If you're working on foundational elements then literally nothing has changed. If you're shipping products then absolutely retool around a cloud provider, the infra isn't your secret sauce and if you have to move back on prem because of cost it will be good problems to have.

I mean this completely sincerely, take a job at a company that's providing hosting/services to people. All the old timers with deep deep systems knowledge are gods.

Re: AWS down again?

#162

Post by [deleted] »

koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

[deleted]

Re: AWS down again?

#163

Post by mac-chaffee »

Flankk wrote: post #103
PedroBatista wrote: post #43

Only the most informed and rational companies will see how and why most of this "in the cloud" thing is a bad idea. Most will accept it as a fact of life and continue to pay for it both directly and indirectly, as long as there's cheap money going around the cloud business can't do wrong. But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.

Are you arguing that purchasing and maintaining your own servers is cheaper than using AWS? Let's hear the rationale, then.

Yes, on-prem is generally cheaper in the long-run.

For example, my company recently purchased $400k worth of hardware that would cost $80k/month if it was all EC2. There's amortized costs of the supporting infra (cooling, power) and ongoing maintenance/depreciation, but paying for the hardware in 5 months can't be beat.

Re: AWS down again?

#164

Post by PragmaticPulp »

koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

> I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work.

Engineers like to build stuff. Historically, running your own infrastructure and configuring servers from the ground up were the building blocks of the internet. These new cloud services took away those building blocks and they had the nerve to charge for it.

In the real world, there are more interesting problems to solve than setting up and maintaining your own servers so it isn't really a complaint. It's actually more fun to get the hosting stuff out of the way and focus on the problem at hand.

HN comments are basically notorious for exaggerating the downsides of hosted services while overplaying the ease and benefits of DIY. A lot of the comments here are similar to the famous Dropbox comment thread where users couldn't understand why anyone would want to use Dropbox when they could simply set up a complicated self-hosted rsync contraption to sync their own files.

Re: AWS down again?

#165

Post by tuldia »

ksec wrote: post #153
technobabbler wrote: post #104

Earlier quoted context omitted.

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

>making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

> You can now simply point to AWS is down

Heck, if that was even possible... "everything is green" dashboards... :-)

Re: AWS down again?

#166

Post by Am_I_Right »

oxfordmale wrote: post #116
5e92cb50239222b wrote: post #95

Earlier quoted context omitted.

Wha? We host everything on our hardware (which is nothing special) and haven't had any downtime in this year (yet). And we're just another run of the mill dev shop, very far from "superstars" who work on these (supposedly extremely stable) platforms.

Two questions I have regarding your in-house hardware: 1. How easy can I access your physical servers ? 2. What happens if there is a catastrophic failure, for example local power outage or a major flooding 3. How secure is your server? Are you regularly patching your operation systems 4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going…

Not the person you're replying to, but...

1: Access to the data center requires either an access card plus biometric verification (fingerprint in one DC in my case, retina scan in the other), or an ID-verified appointment. Then, you still need to know where my server is, plus the access code for the rack (or, you need to be an average lockpicker, but beware, there's cameras...).

2: Each data center has dual-provider AC feeds, plus generators, and provides A/B feeds to my rack. I've not had a dual-feed outage in the last 20 years or so.

3: No cloud provider that I'm aware of guarantees server security or does automated patching (for servers, not services). So, keeping your server up-to-date seems equally important for both cloud and non-cloud scenarios?

4: At least two weeks, I guess? I have sufficient VM host capacity to accommodate 30% unplanned growth, but 100% would require new hardware. So: ordering two servers, installing these in two data centers. But if the new project also requires significant bandwidth, getting new Internet connections in might take longer.

Look, I'm definitely not denying that "the cloud" makes it easier to scale fast, but scaling fast is not an overriding concern for most businesses. Cost is, and self-hosting, even with a pretty redundant infrastructure, is still much cheaper than AWS.

Re: AWS down again?

#167

Post by gerbilly »

The whole thing reminds me of the early nineties.

There was a whole mature infrastructure built around UNIX and open standards at the time.

We were told to scrap that and replace it with Windows NT and its descendants.

While UNIX wasn't perfect¹, Windows was an order of magnitude worse.

Compared to our UNIX machines, those systems got the internet protocols wrong, were full of security holes and crashed all the time. Objectively worse performance and security was tolerated for a decade or more, just because we all convinced each other that this was the way things were going.

It took Microsoft maybe ten fifteen years to work their problems out. And now there's a Linux env built into Windows.

This cloud thing isn't the end of computing history, just like Windows on the server wasn't the end of history.

Cloud isn't better than on prem. It's just popular right now.

1: https://en.wikipedia.org/wiki/Morris_worm

Re: AWS down again?

#168

Post by gtirloni »

gerbilly wrote: post #155
gtirloni wrote: post #124

Earlier quoted context omitted.

Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.

False dichotomy, if you say so. I haven't found it too helpful when dealing with AWS and the serverless trend, whose popularity is really just based on price and economics, not technical superiority. Serverless turns every simple system into a distributed system with the number of failure modes now multiplied by ten. That sounds like fun. I do know how to use serverless for the record, I just think it's an overhyped,…

You just moved the goal post from "the cloud" to "serveless". Anyway, this discussion is full of overgeneralizations that are not useful. I'm abstaining from further comments.

Re: AWS down again?

#169

Post by api »

carlmr wrote: post #121
api wrote: post #93

Earlier quoted context omitted.

This is particularly true with modern fully solid state hardware. Uptimes of decades are likely possible if you don't mess with anything. We run some bare metal servers. They just never go down. Solid continuous pings for years as monitored from elsewhere on the Internet. That's because they're just boxes on a rack somewhere running an OS and some steady-state services (ZeroTier roots). Simplicity is more robust than…

>Many companies spend enough on AWS to buy an entire rack of hardware at a different data center every month and pay 2-3 employees to manage it. That hardware would be up to 100X as fast and powerful as what they rent at AWS and bandwidth would be almost free. That's a really distorted market. The amortized costs should not be this different. I think the issue here is that it's not zero-cost to switch. Your processes…

Things like OpenStack are promising if they can become easier to install and run.

https://www.openstack.org

Re: AWS down again?

#170

Post by tyingq »

willcipriano wrote: post #111
papito wrote: post #57

Earlier quoted context omitted.

Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.

Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.

It seems like they could at least cordon some of it off into a corral of "legacy products". Or maybe a concept of "next gen" regions that have only a subset of products.
Post reply on HN