Tell HN: AWS appears to be down again • Hacker News Board

Re: Tell HN: AWS appears to be down again

#621

Post by AndyJames » Thu, Dec 23, 2021, 9:50 AM UTC

hughesjj wrote: post #596

dylan604 wrote: post #407

Earlier quoted context omitted.

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated…

I thought running a generator full time was illegal AF due to environmental regulations?

Not really. If there's no local law against it then it's legal especially outside the cities.

Re: Tell HN: AWS appears to be down again

#622

Post by sfoley » Thu, Dec 23, 2021, 10:05 AM UTC

Hippocrates wrote: post #198

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not…

Who says this? I have literally never once seen this.

Re: Tell HN: AWS appears to be down again

#623

Post by codeduck » Thu, Dec 23, 2021, 11:34 AM UTC

buchanmilne wrote: post #620

codeduck wrote: post #606

Earlier quoted context omitted.

How are health checks supposed to help when you can't do anything?

You said: > it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands "Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ. If you had instances across multiple AZs behind an ELB with…

You have misunderstood our problem. Ec2 behind elb/alb/nlb is the least of our issues.

Re: Tell HN: AWS appears to be down again

#624

Post by greyhair » Thu, Dec 23, 2021, 1:11 PM UTC

tyingq wrote: post #444

dylan604 wrote: post #407

Earlier quoted context omitted.

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated…

There are setups where the UPS is designed to last long enough for generator spin up as well. I believe it's the most common setup if you have both. I assume spinning up the generators for very short-lived line power blips might be undesirable.

I was in a Bell Labs facility that had notoriously bad power. We occupied the building before the second main feed had been fully approved by the state and run to the building.

Our main computer lab had a serial UPS that was online 100% of the time, though the inverters where under a very light load. If the mains even acted 'weird' (dips, bad power factor, spikes) the UPS jumped full on, and didn't revert to main power until the main was stable for some duration of time. The UPS was able to carry the full lab (which was quite large) for about two hours, allowing plenty of time for the generator to fire up.

The UPS ran a lot, and because the main was 'weird', the outages were often short, the generator wouldn't even start during the first ten minutes of UPS coverage. Of course, the rest of the building would be dark, other then emergency lighting.

I was a embedded firmware engineer, and our development lab was directly on the wall behind the UPS. When it fired into 100% mode, it roared, mostly from cooling. It was sort of a heads up that the power was likely to fail soon.

Re: Tell HN: AWS appears to be down again

#625

Post by BowBun » Thu, Dec 23, 2021, 1:50 PM UTC

doublerabbit wrote: post #248

ryanbrunner wrote: post #193

Earlier quoted context omitted.

I think most people on here are coming from the perspective of startups, which scale out of a single server setup pretty quickly. At a bare minimum, most will have dedicated purpose-built servers like Redis or a DB, and often there's separate background workers, or a load balancer with a couple of web servers. When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first…

> running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount. What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required. If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challen…

You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines. That's a $100K+ job at any reasonable startup (not counting benefits) which is more than our Heroku spend (4-5 year startup). Keep in mind Heroku is way more expensive than just AWS.

This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for. All to solve... a couple days of service-specific downtime a year? People need to chill out with their non-mission critical software. God forbid someone can't access their HR portal for an hour.

Re: Tell HN: AWS appears to be down again

#626

Post by doublerabbit » Thu, Dec 23, 2021, 2:20 PM UTC

BowBun wrote: post #625

doublerabbit wrote: post #248

Earlier quoted context omitted.

> running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount. What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required. If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challen…

You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines. That's a $100K+ job at any reasonable startup (not counting benefits) which is more than our Heroku spend (4-5 year startup). Keep in mind Heroku is way more expensive than just AWS. This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized…

> You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines.

No. You rack the server, connect the cable to the switch press the on-switch, let it boot and then operate as you would with any other computer. Need a new server? Get the DCOps remote hands to Rack the server, cable it to the switch and press the on-switch. If you can install Windows 10, you can install Linux.

> This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for.

Hmm, the data is yours; You own the data and that's value to me if I am the customer. And there's no value to the customer if you were to host it in the cloud either, much of a loss retrospectively.

There is no specialized skill you have to pay for, sure if you were going to run a high-density compute mainframe running some specialized OS like AIX then yeah sure. To buy a server, install a OS and plug it in to a switch and navigate it with SSH requires no overqualified anything.

Re: Tell HN: AWS appears to be down again

#627

Post by lambic » Thu, Dec 23, 2021, 2:29 PM UTC

manquer wrote: post #455

lambic wrote: post #422

Earlier quoted context omitted.

We have AWS backups, "offsite" backups on another cloud provider, and air-gapped backups in a disconnected hard drive in a safe. The extra expense outlay for the 2 additional backups is approximately $50/month, so it's not going to break the bank.

Egress from aws is not cheap. At $50/month scale a lot of things are possible. Most companies cannot store their data in a hard disk in a safe. If you can, then cloud is a convenience not a necessity for you. I.e. you are perfectly fine running your storage stack for the most part. My company is not very big(100ish employees) and we pay $200k+ for AWS in just storage and AWS is not even out primary cloud. If we have…

> My company is not very big(100ish employees)

I don't get how this is relevant at all, it's more about how much data your company stores than how many employees it has.

I've worked for a company with 5000 employees that stored less data (fewer data?) than my current employer that has less than 100.

> No sane business is going to sign off on +3x storage costs on a risk that cannot be easily modeled

Probably not, but for us the cost is about 0.1x our aws storage costs, so it's a no-brainer.

Re: Tell HN: AWS appears to be down again

#628

Post by doublerabbit » Thu, Dec 23, 2021, 2:29 PM UTC

manquer wrote: post #365

doublerabbit wrote: post #248

Earlier quoted context omitted.

> running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount. What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required. If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challen…

Long term is unknown to the startup , they may fail or pivot . Buying upfront hardware is not feasible even if I had the cash(which most don't), I don't know if the company would last that long or would be doing things that require x servers . What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect…

> Buying upfront hardware is not feasible even if I had the cash(which most don't)

You only need one server, slap on a hypervisor and your rocking. Heck you can buy entry servers from dell for budget; upgrade later.

A 10u rack, which is adequate for any small business comes to around $500 a month in LA. 4u would be more than enough for a startup and that's around $200/month.

Is the start-up not going to purchase computer hardware, monitors, television screens for their clients when they sit in the waiting room? Email accounts with Office365, a website, a domain name? If they can fit that in to their budget I am pretty sure they could afford a server and colocation space.

> What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.

But colocation is dynamic. Contracts can be negotiated.

> buy into rent seeking business models

And AWS isn't a rent seeking business model?

Looking at EC2 instances, for 120GBHD, 32 cores "Dedicated" instance, your looking at 679.54 USD for a month. 120GB isn't much especially when the developers start doing their thing.

For $500 you can have so much more, and hardware you actually own and that if the company does not lift off it can then be sold. Is that not the better investment?

No remarks to lack of skills.

Re: Tell HN: AWS appears to be down again

#629

Post by BowBun » Thu, Dec 23, 2021, 2:45 PM UTC

doublerabbit wrote: post #626

BowBun wrote: post #625

Earlier quoted context omitted.

You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines. That's a $100K+ job at any reasonable startup (not counting benefits) which is more than our Heroku spend (4-5 year startup). Keep in mind Heroku is way more expensive than just AWS. This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized…

> You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines. No. You rack the server, connect the cable to the switch press the on-switch, let it boot and then operate as you would with any other computer. Need a new server? Get the DCOps remote hands to Rack the server, cable it to the switch and press the on-switch. If you can install Windows 10, you can install Linux. >…

Configuring a server for high traffic, networking, various safety measures against fire/power, backups, connectivity to your external DBs, etc. etc. As a 'feature dev' of almost a decade, I could bungle my way through this and probably leave a bunch of security gaps open. I'm highlighting the fact that it is a specialized skill and a specific part of the stack that feature devs like me and many other early startup engineers forego for other skills.

To your second point: We still own the data for hosted solutions. We have our backups, we have direct access. Barring a catastrophic failure and wiping of AWS, our data is there and ours. The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes. And yes I know the argument of "you end up spending more time debugging AWS", but I think you don't need to reach that point if you keep things pretty simple, especially early stage.

I think you're vastly oversimplifying the task. And if I had to guess, it's something you're pretty familiar with so it makes sense that it's easy to you! I'm sure an early stage company would be happy to have you to save a ton of money in their early days on infrastructure costs

Re: Tell HN: AWS appears to be down again

#630

Post by ctvo » Thu, Dec 23, 2021, 3:43 PM UTC

iso1631 wrote: post #567

ctvo wrote: post #240

Earlier quoted context omitted.

> Not at this rate. And what rate is this? It gets attention because it impacts more people, but AWS / GCP / Azure uptime is still better than what I've seen for small / mid size businesses trying to manage their own infrastructure.

Multiple outages in a single month.

Across different regions, availability zones, and services. What's the impact to customers? Today it was a very specific EC2 outage in an AZ in us-east-1.

So again, we're talking about cloud providers because of their scope and size, and they're still doing better than MidSizedBank managing their own infrastructure.