Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

511–520 of 646 posts

Re: Tell HN: AWS appears to be down again

#511

Earlier quoted context omitted.

> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…

My point is if your redundancy is better than AWS then why pay for them ? If it not they why invest in your own?. You can argue that you protect against different threats than AWS does . So far I have not seen a meaningful argument of threats a on Prem protects differently than the cloud that you need both . Say for example your solution is to put all your data backups on the moon then it makes sense to do both, AWS…

> Even Amazon.com or Google apps host on their own cloud and not use multi cloud after all, their regular businesses are much bigger than their cloud biz

This is probably true with Google, but AWS contributes > 50% of Amazon's operating income. [1]

[1] https://www.techradar.com/news/aws-is-now-a-bigger-part-of-a...

Re: Tell HN: AWS appears to be down again

#512

Earlier quoted context omitted.

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

It is much easier than you think. There are well-defined standards and trained trades people and whole host of companies who make great products and provide after-sales services to do it. Every major financial services, telecom and high-precision manufacturing companies run their infra this way. It is definitely less niche than rolling your own encryption.

    financial services, telecom and high-precision manufacturing companies
One of these things is not like the other, one of these things is not the same...

What use does a CNC shop have for an extensive on-prem multi-DC with failover and high availability? It'd be like buying your own snowplows to make sure that the road is clear so your employees can get to work. Maybe necessary if you live in a place with very bad snowplows and no existing infrastructure, but in most places, just a waste of money.

Re: Tell HN: AWS appears to be down again

#513

Earlier quoted context omitted.

That's true for on-prem infrastructure, but is all already handled for you if you rent servers from hosting providers such as OVH/Hetzner or even rent colocation space in an existing DC, and is still cheaper than the cloud equivalent (and as we saw recently, actually more reliable as well).

I've had way more networking and availability failures from Hetzner this year alone then I've ever seen from AWS. They regularly replace their networking switches without any redundancy, leaving entire DCs offline for hours. They're okay for hobby projects, but I would never host a business-critical site with them

Cannot confirm, do you have details?

Yes, Hetzner upgrades DCs (datacenter buildings), but they are the equivalent to AWS AZs (Availability Zones). When they upgrade a DC, they notify way in advance, and if you set up your services to span multiple DCs as is recommended, it does not affect you.

We run high-availability Ceph, Postgres, and Consul, across 3 Hetzner DCs, and have not had a Hetzner-induced service downtime in the 5 years that we do so.

Re: Tell HN: AWS appears to be down again

#514

Earlier quoted context omitted.

Your us-east-1b may be the parents us-east-1c. The letters are randomised per AWS account so that instances are spread evenly and biases to certain letters don't lead to biases to certain zones.

Huh, that's interesting. Didn't know that, but makes sense.

You can check which availability zone is with: aws ec2 describe-availability-zones --region us-east-1

Re: Tell HN: AWS appears to be down again

#515
post #473

Earlier quoted context omitted.

My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, eve…

Indeed, if you only deploy resources in us-east1, or any other single region, you're risking the occasional downtime. I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.

Honestly, I have an app in production that isn't completely hardened against single zone outages. There was pressure to turn off some redundancy in our caching infra, and not every backend service we call is free of tenant affinity, so we could well lose at least 1/3rd of our customers in a single AZ failure in the wrong region, or have huge latency issues for all of our tenants based on high cache miss rates.

Having written this, I'm going to ping our SME on the cache replication and remind him that since the last time he benchmarked it, we've upgraded to a newer generation of EC2 instances that has lower latency, and could he please run those numbers again.

Re: Tell HN: AWS appears to be down again

#516
post #471

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

I had this experience when I asked about s3 backup also (after a junior programmer deleted a directory in our s3 bucket...). The response from r/aws was "just don't let that happen" or ("use IAM roles")

411, in the latest reInvent AWS announced preview of AWS Backup for S3 (right now in USW2 only).

Relevant blog post, https://aws.amazon.com/blogs/aws/preview-aws-backup-adds-sup...

Re: Tell HN: AWS appears to be down again

#517
post #366

Earlier quoted context omitted.

> The facilities team was about 6 people, average salary around $90k. Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs: - Annual salary increases - Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems) - Systems that run in the data center (logging, monitoring, etc.) - Procurement costs with changing costs in hardware (silicon shortages, etc.) - Security co…

Say all of those costs are $2,000,000, and you have 25,000 billable endpoints in the datacenter… you’re looking at less than $0.01/hour for that overhead on a unit basis. Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity. Just to be clear, I’m not arguing that on-prem is magically cheap. :)…

Agreed on all accounts.

Re: Tell HN: AWS appears to be down again

#518
post #258

Earlier quoted context omitted.

Omg, this needs to be on a plaque or something. "Let's move our internal app with 50 users to k8s in the cloud." --true story

Amazing. And as long as "technological progress" sufficiently obscures the impact of such ridiculousness the more such projects will continue to occur. It's a real shame that the collective world of technology does not properly respect the simple solutions that work. It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those…

Yeah, after getting caught up in the hype for ten years I'm running back to proven tech that is flushed out (Java, Swing-omg it just works, wanting to try Ruby,even PHP is looking good at this point).

Every dependency scrutinized and discarded if possible.

I would probably work for free if someone setup their own on-prem cloud in Tanzu, Open shift, or Ranger and used old school proven frameworks for development.

Working in AWS has been a real shitty experience at these large companies. All the nit picky problems (of which there are thousands) get dumped on devs who are trying to deliver working software.

Re: Tell HN: AWS appears to be down again

#519

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

Would you put the one and only copy of your family photo album up on AWS, where AWS going down would mean losing it? Because your customers' data is more important than that

Re: Tell HN: AWS appears to be down again

#520
post #472

Earlier quoted context omitted.

1 - For the last 10 years, servers have been beasts. You have a lot of cores, plenty of HD and RAM. Servers are less expensives than devs. Scaling vertically can go VERY far. 2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary f…

This was super interesting to read, thank you very much. Regarding the ffmpeg parameters and formats in general: Do you use newer formats too, like AV1 and the like?

No, we use only H264, because nobody has the courage to redo all the work we've done to optimize the encoding with a newer format :)
Post reply on HN