Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS status updates not working due to S3

twitter.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: AWS status updates not working due to S3

#41

Post by ryanbrunner »

alpb wrote: post #34

> The dashboard not changing color is related to S3 issue. I don't understand this. The icon URL is in the HTML. Both icons https://status.aws.amazon.com/images/status0.gif and https://status.aws.amazon.com/images/status3.gif have been working for us all along. Plus clearly they are able to update the status page contents, because they added the "increased error rates" message there too. I don't want to believe it bu…

One explanation might be that they use an internal tool to update the status page definitions, and parts of that tool are hosted on S3. Or that the status definitions themselves are hosted on S3 (and then read and transformed into the HTML page everyone sees)

Re: AWS status updates not working due to S3

#42

Post by simplehuman »

rifung wrote: post #40
simplehuman wrote: post #20

Earlier quoted context omitted.

> Writing a B+ tree from memory and making sure your infrastructure isn't doing something stupid are fundamentally different skills. It's funny. You know it. I know it. Entire HN knows it. And yet _no_ interview follows any such common sense rules. Just go to a Google/FB interview and they ask you all sort of questions. It doesn't matter what you are interviewing for. In fact, in many cases they don't even tell you w…

> Since they will "assess" where you fit best. Disclaimer: I work for Google At Google this just isn't true unless we are talking about new grads. After you pass the interviews, you have to do team matching, where you will have informal 2 way interviews with prospective teams. Only after you find a team that you like and that likes you can you get an offer, assuming your application is approved. It's definitely not a…

Thanks for the reply. I am talking about the interviewing stage. I have more than 10 years of engineering experience but the Google HR actually sent me a PDF of all the topics I should be well versed it. The booklet was basically my entire grad course and masters and more. I can confirm from more than 5 sources that this is the case with Google interviews.

Re: AWS status updates not working due to S3

#43

Post by simplehuman »

kevan wrote: post #35
simplehuman wrote: post #4

It baffles me that AWS, a leader in cloud computing can make such a rudimentary mistake. Seriously, I interviewed there and they asked me to write a b+ tree and I failed. And then you see fundamental errors like this which possibly cannot be made by people who had the smarts to write b+ trees in 15 minutes... I want to take this opportunity to complain about the interview system. Hire people who care about the produc…

>Such mistakes cannot be made by people who care. That's a very naive assertion. Humans make mistakes, they always have and they always will, no matter how smart they are and how much they care. That's why pilots have checklists that they go through before they're even allowed to leave the gate.

This statement (it starts with 'such') was about a very specific mistake. This is not some hard engineering problem. People who 'care' about the authenticity of the status page will not make the error of basing it on the infrastructure it monitors.

Re: AWS status updates not working due to S3

#44

Post by jaredsohn »

simplehuman wrote: post #42
rifung wrote: post #40

Earlier quoted context omitted.

> Since they will "assess" where you fit best. Disclaimer: I work for Google At Google this just isn't true unless we are talking about new grads. After you pass the interviews, you have to do team matching, where you will have informal 2 way interviews with prospective teams. Only after you find a team that you like and that likes you can you get an offer, assuming your application is approved. It's definitely not a…

Thanks for the reply. I am talking about the interviewing stage. I have more than 10 years of engineering experience but the Google HR actually sent me a PDF of all the topics I should be well versed it. The booklet was basically my entire grad course and masters and more. I can confirm from more than 5 sources that this is the case with Google interviews.

You stated two things: the fact that Google asks algorithms questions and "Since they will "assess" where you fit best". The GP was refuting the second of these things (the GP even quoted it) while you were defending the first statement.

BTW, I think that second point is a common misconception that deserves rebuttal because I think that until a five years ago or so Google wouldn't tell you which team you would join (or give you much choice) before deciding on an offer, even if experienced.

Re: AWS status updates not working due to S3

#45

Post by kevan »

simplehuman wrote: post #43
kevan wrote: post #35

Earlier quoted context omitted.

>Such mistakes cannot be made by people who care. That's a very naive assertion. Humans make mistakes, they always have and they always will, no matter how smart they are and how much they care. That's why pilots have checklists that they go through before they're even allowed to leave the gate.

This statement (it starts with 'such') was about a very specific mistake. This is not some hard engineering problem. People who 'care' about the authenticity of the status page will not make the error of basing it on the infrastructure it monitors.

Takeoff in a plane isn't a hard engineering problem either, but without checklists even one of the best pilots in the world forgot to unlock his rudder before heading down the runway. [1]

Of course no one who cares would intentionally make the choice of building the status page on the infrastructure that it monitors, but it's not that difficult for something to creep into the dependency chain and then you don't find out until the next outage a year later.

Mistakes were made, but attributing it to lack of caring is misguided. Good intentions don't work, you need mechanisms to enforce good practices. Those mechanisms obviously failed here, but the solution is to fix the mechanisms, not ask people to care more.

[1] http://www.ww2hc.org/emailarchives/2011/checklistorigin.htm

Re: AWS status updates not working due to S3

#46

Post by burntrelish1273 »

simplehuman wrote: post #4

It baffles me that AWS, a leader in cloud computing can make such a rudimentary mistake. Seriously, I interviewed there and they asked me to write a b+ tree and I failed. And then you see fundamental errors like this which possibly cannot be made by people who had the smarts to write b+ trees in 15 minutes... I want to take this opportunity to complain about the interview system. Hire people who care about the produc…

Because bureaucracy, siloization, Tragedy of the Commons and likely a massive infrastructure with tons of technical debt and inability to make major changes except by gradual incrementalism, oft too late. Infrastructure needs active SimianArmy-style breakage finding without "sacred cows" but with less than 100% uptime across all services to ferret-out outage edge-cases.

Re: AWS status updates not working due to S3

#47

Post by burntrelish1273 »

simplehuman wrote: post #20
wheaties wrote: post #15

Earlier quoted context omitted.

Writing a B+ tree from memory and making sure your infrastructure isn't doing something stupid are fundamentally different skills. One requires that you regurgitate the contents of a text book on a white board, the other that you can engineer a solution. I wish them well on an interview set up for hiring the former; I try to hire the later.

> Writing a B+ tree from memory and making sure your infrastructure isn't doing something stupid are fundamentally different skills. It's funny. You know it. I know it. Entire HN knows it. And yet _no_ interview follows any such common sense rules. Just go to a Google/FB interview and they ask you all sort of questions. It doesn't matter what you are interviewing for. In fact, in many cases they don't even tell you w…

Lazy interviewers looking for magic shortcuts / lazy managers not mentoring/teaching/structuring how to dig into candidate ability to solve real, hairy problems quickly and get a window into candidate problem-solving thought-processes. Hiring by random "intuition" is both evidence-free and scatter-gun half-assery that results in wasting everyone's time by not attracting/selecting for great candidates.

Re: AWS status updates not working due to S3

#48

Post by burntrelish1273 »

gdulli wrote: post #36
simplehuman wrote: post #4

It baffles me that AWS, a leader in cloud computing can make such a rudimentary mistake. Seriously, I interviewed there and they asked me to write a b+ tree and I failed. And then you see fundamental errors like this which possibly cannot be made by people who had the smarts to write b+ trees in 15 minutes... I want to take this opportunity to complain about the interview system. Hire people who care about the produc…

1. The number of people at Amazon who need to be able to write a b+ tree is negligible. 2. The number of people at Amazon who need to be able to choose a search/sort/etc. algorithm or data structure and understand why one is more appropriate than another for a given use case is much higher. 3. The number of people at Amazon who need to demonstrate common sense is very high. This skill is much more closely related to…

Yup, it's BS in-lieu of real problems.

CS fundamentals are nice to know, but how often does one implement something custom like BigTable/Colossus from scratch vs. buy/use OTS? The support/scalability/technical debt/unforeseen costs of implementing something entirely new is typically much greater than using adequate "lego" that already exist.

Judgement of cost/benefit DIY vs. OTS can be gained (hopefully) without too much wasted effort, time, money, morale & business life-expectancy.

Re: AWS status updates not working due to S3

#50

Post by manojlds »

asp2insp wrote: post #22
simplehuman wrote: post #10

Earlier quoted context omitted.

And yet, I cannot find any obvious information on where statuspage is hosted.

Builtwith seems to think they are hosted on EC2 https://builtwith.com/statuspage.io

Their status page says so anyway - http://metastatuspage.com/incidents/lb3rpt031vmx
Post reply on HN