Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

401–410 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#401

Post by oars »

belter wrote: post #68
ALittleLight wrote: post #48

Earlier quoted context omitted.

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

Was this Amazon or AWS?

Based on the other comments in this comment thread I would say it's Amazon.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#402

Post by log4shell »

EdwardDiego wrote: post #167
shagie wrote: post #160

Earlier quoted context omitted.

Consider the possible ways for a string to be injected into any of the following: Apache Solr Apache Druid Apache Flink ElasticSearch Flume Apache Dubbo Logstash Kafka If you've got any of them, they're likely exploitable too. That list comes from: https://unit42.paloaltonetworks.com/apache-log4j-vulnerabili... The attack surface is quite a bit larger than many realize. I recently had a conversation with a person who…

Kafka is still on log4j1. It's only vulnerable if you're using a JMSAppender.

Kafka is not vulnerable for this particular exploit but hdfs kafka connect plugin is.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#403

Post by DrBenCarson »

bradknowles wrote: post #392
DrBenCarson wrote: post #125

Earlier quoted context omitted.

I'm going to leave it at this: the dashboards at AWS aren't automated. Say what you will, but I can automate a status dashboard in a couple days--yes, even at AWS scale. No reason the dashboard should be green for hours while their engineers and support are aware things aren't working.

Uh, no. You can’t. If you could, then you would already have been hired and you would have already solved this problem. What you can do at what you think is AWS scale has no bearing on what you could actually do at real AWS scale.

Right I guess I would be hired without acknowledging my recruiter or applying I guess that makes sense

Related: Are you familiar with log aggregation/streaming?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#404

Post by efitz »

Karunamon wrote: post #215
efitz wrote: post #199

Earlier quoted context omitted.

You don’t know what you’re talking about. AWS spends a lot of time thinking about this problem in service to their customers. How do you reduce the status of millions of machines, the software they run, and the interconnected-ness of those systems to a single graphical indicator? It would be dumb and useless to turn something red every single time anything had a problem. Literally there are hundreds of things broken…

> How do you reduce the status of millions of machines, the software they run, and the interconnected-ness of those systems to a single graphical indicator? A good low-hanging fruit would be, when the outage is significant enough to have reached the media, you turn the dot red . Dishonesty is what we're talking about here. Not the gradient when you change colors. This is hardly the first major outage where the AWS st…

Human-in-the-loop != lying.

Broken dashboard != lying.

The specific charge of “lying” is what I dispute.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#405

Post by ranguna »

tuldia wrote: post #324
ranguna wrote: post #312

Earlier quoted context omitted.

You can either pay a dedicated team to manage your on prem solution, go multi cloud, or simply go multi region on aws. My company was not affected by this outage because we are multi region. Cheapest and quickest option if you want to have at least some fault tolerance.

> ... multi region. Cheapest and quickest option if you want to have at least some fault tolerance. That is simple not true, you have to adapt your application to be multi region aware to start with, and if you do that on AWS you are basically locked-in, and one of the most expensive cloud providers out there.

You're saying it's not true, but do you have another example of a quick and cheap way to do this ?

I'm not saying this can be done in 1 day for 2 cents, I'm saying that it's quick and cheap compared to other options.

> adapt your application to be multi region aware

This vs adapting your application to support multi cloud deployments or go from the cloud to start doing on prem with a dedicated team, you can take your bets.

On aws you can setup route 53 to point to multiple regions based on health check or latency.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#406

Post by Ensorceled »

ithkuil wrote: post #349
Ensorceled wrote: post #338

Earlier quoted context omitted.

> Outrage is the easy response. Empathy and learning is the valuable one. I'm outraged that AWS, as a company policy, continues to lie about the status of their systems during outages, making it hard for me to communicate to my stakeholders. Empathy? For AWS? AWS is part a mega corporation that is closing in on 2 TRILLION dollars in market cap. It's not a person. I can empathize with individuals who work for AWS but…

My reading of GP's comment is that the empathy should be directed towards AWS' team , the people who are building the system and handling the fallout, not AWS the corporate entity. I may be wrong, but I try to apply the https://en.m.wikipedia.org/wiki/Principle_of_charity

The very first line of the comment is:

> Complex systems are really really hard. I'm not a big fan of seeing all these folks bash AWS for this,

The rest is support for the organization and the complexity of their product offering.

The only line that is about supporting or empathy for the staff is the last sentence:

> Hugs to the AWS team, and good learnings for everyone.

I don't think I'm out of line or stretching here.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#407

Post by discodave »

wbsun wrote: post #211
discodave wrote: post #103

Earlier quoted context omitted.

I saw pleeeeeenty of untested code at Amazon/AWS. Looking back it was almost like the most important services/code had the least amount of testing. While internal boondoggle projects (I worked on a couple) had complicated test plans and debates about coverage metrics.

Interesting. I also work for a cloud provider. My team work on both internal infrastructure as well as product features. We take testing coverage very seriously and tie the metrics to the team's perf. Any product feature must have unit tests, integration tests at each layer of the stack, staging test, production test and continuous probers in production. But our reliability is still far from satisfactory. Now with yo…

> Now with your observation at AWS, I start wondering whether the coverage effort and different types of tests really help or not...

Figuring out ROI for testing is a very tricky problem. I'm glad to hear your team invests in testing. I agree it's hard to know if you're wasting money or not doing enough!

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#408

Post by discodave »

jvolkman wrote: post #191
discodave wrote: post #109

Earlier quoted context omitted.

It's 2006, you work for an 'online book store' that's experimenting with this cloud thing. Are you going to build a whole new network involving multi-million dollar networking appliances?

Developing EC2 did involve building a whole new network. It was a new service, built from the ground up to be a public product.

Yes, but many services, including S3, SQS, and DynamoDB did not run on the EC2 network for most of their existence.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#409

Post by discodave »

soheil wrote: post #204
discodave wrote: post #109

Earlier quoted context omitted.

It's 2006, you work for an 'online book store' that's experimenting with this cloud thing. Are you going to build a whole new network involving multi-million dollar networking appliances?

No but one would hope 15 years and 1 trillion dollars later you would stop running it on the computer under your desk.

Hehe true! But it's real hard to "move fast and not break things" when you're talking about millions of servers, exabytes of data, and tens of thousands of engineers.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#410

Post by jvolkman »

discodave wrote: post #408
jvolkman wrote: post #191

Earlier quoted context omitted.

Developing EC2 did involve building a whole new network. It was a new service, built from the ground up to be a public product.

Yes, but many services, including S3, SQS, and DynamoDB did not run on the EC2 network for most of their existence.

The netscalers were on prod after all.
Post reply on HN