AWS down again? • Hacker News Board

Re: AWS down again?

#221

Post by stjohnswarts » Fri, Dec 10, 2021, 4:58 PM UTC

willcipriano wrote: post #111

papito wrote: post #57

Earlier quoted context omitted.

Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.

Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.

I think the term is fine. I don't don't know if is the "ever accumulating" version is correct as at some point it levels off in my experience. At some point things break and you have to take care of it.

Re: AWS down again?

#222

Post by int0x2e » Fri, Dec 10, 2021, 5:04 PM UTC

tres wrote: post #172

julianlam wrote: post #132

Earlier quoted context omitted.

That's an interesting position to take, because I imagine if a team of developers decided to invent the internet in a vacuum today, it'd be a hell of a lot more secure than the "let's hope nobody uses this protocol maliciously" attitude prevalent in the early days of the internet. Not that that is a bad thing, but just something to think about.

It wasn't an attitude of naive hope that there wouldn't be nefarious actors leveraging the protocol; there was a different kind of people using the Internet. There's no need to design security in the system when you literally know everyone who is using it. And everyone who was using it had the same goals in mind. So, I don't disagree with the sentiment -- people today would probably do it a little bit differently; ho…

While I don't disagree, if people at the time had assumed that everyone on the network could be trusted (forever), why design the IPv4 address space to make room for 4 billion devices? Why support so many ports and concurrent connections? The two assumptions don't quite match up.

Re: AWS down again?

#223

Post by stjohnswarts » Fri, Dec 10, 2021, 5:05 PM UTC

iso1631 wrote: post #76

41209 wrote: post #67

Earlier quoted context omitted.

Would you rather have to fix your own data center, or wait 4 hours. AWS works 99% of the time,plus it's someone else's problem

In my industry, five nines is the starting level. You're proposing something 1,000 times worse.

Then AWS isn't really targeting you as a customer though. Most of the web really isn't that critical and can survive a few hours outage occasionally.

Re: AWS down again?

#224

Post by registeredcorn » Fri, Dec 10, 2021, 5:10 PM UTC

tres wrote: post #172

julianlam wrote: post #132

Earlier quoted context omitted.

That's an interesting position to take, because I imagine if a team of developers decided to invent the internet in a vacuum today, it'd be a hell of a lot more secure than the "let's hope nobody uses this protocol maliciously" attitude prevalent in the early days of the internet. Not that that is a bad thing, but just something to think about.

It wasn't an attitude of naive hope that there wouldn't be nefarious actors leveraging the protocol; there was a different kind of people using the Internet. There's no need to design security in the system when you literally know everyone who is using it. And everyone who was using it had the same goals in mind. So, I don't disagree with the sentiment -- people today would probably do it a little bit differently; ho…

To expand on what's being referenced here, consider the following: video game speedruns.

Throughout the 80s, 90s, and early-to-mid 2000s, there was a certain level of trust in the claims people made about PBs (Personal Bests) and WRs (World Record/Ranking). There was no practical way to record, host, or especially upload literal hours of footage (VHS footage) of a run you did. Even if you did somehow achieve all of the above, it would be a grainy, low quality video which is hard to see, maybe with a stopwatch nearby so people can verify your claim. People would be watching this through RealPlayer, if they could watch it at all!

So what do you do in such a situation where people have no practical or easy means to verify claims? You build credibility off of how active you are with other members of the community. You post and comment on forums about what strategies you're trying, what difficulties you're dealing with, and what new information you might have uncovered through trial and error. You don't prove your work, you prove your worth. Your standing is evidence of your claim.

To me, this is a great example of "personality-credit" communities that's existed online; Usenet and BBS aside. The mentality has largely faded away with improvements to bandwidth and services like Twitch and YouTube, but considering the technological challenges of what someone in say, 1993 would be dealing with in trying to prove they just set a new record can really give a glimpse into what things used to be like.

Re: AWS down again?

#225

Post by oxfordmale » Fri, Dec 10, 2021, 5:21 PM UTC

grey-area wrote: post #147

oxfordmale wrote: post #116

Earlier quoted context omitted.

Two questions I have regarding your in-house hardware: 1. How easy can I access your physical servers ? 2. What happens if there is a catastrophic failure, for example local power outage or a major flooding 3. How secure is your server? Are you regularly patching your operation systems 4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going…

There are many approaches which don't depend on AWS, and not all of them mean hosting your own physical servers, and they certainly don't mean you don't have an off-site backup policy. There are well-understood answers to all your questions, they are not too difficult, they just cost money - some businesses choose not to spend that money, some weigh cost-benefit and go for AWS, some decide to go for in-house servers,…

i am not saying everyone needs to go for AWS and there are many good reasons for self hosting However, if the only reason for self hosting is a higher uptime than AWS or GCP, it is general false economy, and you are likely taking short cuts that you are not aware of.

Re: AWS down again?

#226

Post by stjohnswarts » Fri, Dec 10, 2021, 5:23 PM UTC

Sosh101 wrote: post #26

It seems like AWS and Google have had more outages than normal recently. I guess this is to do with COVID disruptions? (remote working and such).

why would remote workers matter? AWS is basically all remote.

Re: AWS down again?

#227

Post by oxfordmale » Fri, Dec 10, 2021, 5:26 PM UTC

y4mi wrote: post #181

rstupek wrote: post #101

Earlier quoted context omitted.

That is true. But hope a drive by ddos doesn't pick you that day!

How do the attackers ddos an inhouse deployment that cannot be accessed from the internet without joining a VPN? They can try to saturate the VPN host maybe, but that's going to be challenging considering that it's going to be limited to connection requests without valid credentials. and these are likely set to be ignored on multiple failed attempts through fail2ban or similar tooling

Zero Day VPN vulnerabilities

Re: AWS down again?

#228

Post by SahAssar » Fri, Dec 10, 2021, 5:40 PM UTC

ksec wrote: post #153

technobabbler wrote: post #104

Earlier quoted context omitted.

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

>making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

Do people not take responsibility for their choice of vendors? If boeing crashed a bunch of planes and said "Not our fault, our subcontractor made shitty software" would we accept it? If a startup leaked all their customers data would we accept "Blame this other company, we just gave them all our data for analysis" as an excuse?

If you accept that then it means that we need to investigate and scrutinize the whole supply chain of any product before trusting it and it would also mean that any company switching even the smallest part of their supply chain would require us to reevaluate their product.

When we buy things (either products, services or software) we consider the company selling it to be responsible for the whole and deferring responsibility to a vendor is basically just saying that you were either promising more than you knew could deliver or that you did not do a good job picking vendors.

Re: AWS down again?

#229

Post by oxfordmale » Fri, Dec 10, 2021, 5:46 PM UTC

snovv_crash wrote: post #189

organic_popcorn wrote: post #182

Earlier quoted context omitted.

> The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead. Losing skills you worked on for years is just part of this spac…

The entirety of StackOverflow runs on something like 4 machines. Abstraction layers are expensive, and having to learn scaling methodologies unnecessarily when a better choice of upfront technology would render it unnecessary is very un-agile.

StackOverlow doesn't run on just 4 machines. Even in 2016 it required significant hardware:

4 Microsoft SQL Servers (new hardware for 2 of them) 11 IIS Web Servers (new hardware) 2 Redis Servers (new hardware) 3 Tag Engine servers (new hardware for 2 of the 3) 3 Elasticsearch servers (same) 4 HAProxy Load Balancers (added 2 to support CloudFlare) 2 Networks (each a Nexus 5596 Core + 2232TM Fabric Extenders, upgraded to 10Gbps everywhere) 2 Fortinet 800C Firewalls (replaced Cisco 5525-X ASAs) 2 Cisco ASR-1001 Routers (replaced Cisco 3945 Routers) 2 Cisco ASR-1001-x Routers (new!)S

Re: AWS down again?

#230

Post by jjav » Fri, Dec 10, 2021, 5:56 PM UTC

SCHiM wrote: post #21

I think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself). Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political…

> Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages).

Not even so sure about that. I've had a ton more downtime ("degraded" in AWS speak) with AWS than any self-hosted systems. And that's with more than half my career on self-hosted.

If a major disaster strikes, like the whole rack catching fire and melting everything, then it's true that AWS could recover quicker than self-hosted. But most problems are not of that sort.