Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

331–340 of 497 posts

Re: Tell HN: AWS appears to be down again

#331

I'm so glad that I'm not still the CTO of a startup. I would be getting dozens of e-mails from people without engineering backgrounds asking "Are we multi-cloud", "why didn't you make us multi-cloud"?

Well, why didn't you? :)

The response is that this actually works well enough, so the investment required has not pushed anyone to do it (with that meaning building the core infrastructure to make that easy).

Re: Tell HN: AWS appears to be down again

#332

Earlier quoted context omitted.

Surely we are way past the point where someone knows how the whole thing works, all the way down. I doubt even a very skilled engineer would know how his own machine works all the way down. What I think happens mostly is the skilled dev can use his experience to know where to investigate and where to look for solutions. The question is organisational. Might it be that certain orgs have gotten so convoluted that they…

> I doubt even a very skilled engineer would know how his own machine works all the way down Knowing how it works, and being able to build a new one, are also two very different problems. For example, there are plenty of Computer Science folks who learned how to design chips (layout the circuits, write the microcode, etc) - but you need a whole extra background in EE and Physics to be able to fab said chip...

Many programmers complete some kind of nad2tetris[1] style course where they go from basic hardware primitives (the NAND gate) all the way up through a small von Neumann architecture computer that can be programmed with a simple homebrew machine code. Even if they don't, a good CS undergraduate program should cover a lot of it, and since most EEs can program at least a little they probably get a pretty good top-to-bottom understanding as well.

The problem is that this is really only possible with a toy model of a computer and very simple programs. Modern chips with their branch prediction and caching and threading and advanced vectorized operations and so on are vastly more complex. The 6502[2] was perhaps the last chip that one person could fully grok. Maybe a chip designer at Intel or AMD could understand the whole circuit in detail but no one else has the time - it would literally be a full time job. The same thing is true for operating systems - even if you're Raymond Chen, you can know a lot about Windows, but you can't know everything.

We learn just enough about the other parts of the system to convince ourselves that we understand the principles. We build the basic mental model we need to interact with other systems but all we can really do focus on our own specialized areas and hope that everyone else is doing their job. This works well enough until something like Spectre[3] or Meltdown[4] crops up and that's when we realize that we've been building castles in the sand.

[1]: https://www.nand2tetris.org/

[2]: https://en.wikipedia.org/wiki/MOS_Technology_6502

[3]: https://en.wikipedia.org/wiki/Spectre_(security_vulnerabilit...

[4]: https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...

Re: Tell HN: AWS appears to be down again

#333

Earlier quoted context omitted.

Surely we are way past the point where someone knows how the whole thing works, all the way down. I doubt even a very skilled engineer would know how his own machine works all the way down. What I think happens mostly is the skilled dev can use his experience to know where to investigate and where to look for solutions. The question is organisational. Might it be that certain orgs have gotten so convoluted that they…

> Surely we are way past the point where someone knows how the whole thing works, all the way down. I've met a few people who can rightfully lay claim, but yeah, an incredibly rare set of skills. That said, there is a recent revival in building systems from the ground up. While you can't manufacturer your own transistors, it is quite possible to understand everything from simple logic gates to ALUs to older style CPU…

I built a toy CPU in software once as an exercise. I started with "class Transistor" (wrapping an AND op) and "class Wire" (wrapping a boolean), and wired them together incrementally to make gates, flipflops, registers, etc.

I eventually got a fully-functioning 32-bit cpu with instruction pipelining, two levels of cache, DMA input/output, an asynchronous bus, a custom assembly language with an assembler written in python, and got the Game of Life running on it.

It ran about 2kHz with 8kb of memory or so.

Re: Tell HN: AWS appears to be down again

#334

Earlier quoted context omitted.

I do contracting dev work and my specialty is being able to drill down into any part of the engineering assets, ops, sec, dev. People think someone like me is slow and expensive until they have a problem that no one else wants to touch.

> Surely we are way past the point where someone knows how the whole thing works, all the way down. So you are disagreeing with this statement and saying that, in fact, you are the person who knows how the whole thing works? I knew tech work produced some large egos, but sheesh.

This type of person exists, and while rare, not as rare as some seem to assume.

Re: Tell HN: AWS appears to be down again

#335
An honest question. Why do you guys use AWS instead of dedicated servers? It's terribly expensive in comparison, nowadays equally complex, scalability is not magic and you need proper configuration either way, plus now the outages become more and more common. Frankly, I see no reason.

Re: Tell HN: AWS appears to be down again

#336
post #85
post #30

I checked their health status page. All is good. /s https://downdetector.com/status/aws-amazon-web-services/

They did add an update, faster than last time: "7:42 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region." https://status.aws.amazon.com/ Edit: They added US-WEST-1: "7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region." Edit: Found root case, maybe? "8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and ha…

Practice makes perfect

Re: Tell HN: AWS appears to be down again

#337

They really need to stop requiring SVPs or higher to show non-green status on the status page, as other HNers have revealed in last week's AWS post. It's effectively not a status page, and they could probably be sued if it can be demonstrated that X service was down but the status page showed green (since the SLA is based on status page). Should be automated and based on sample deployments running in every region and…

They were much faster than usual about updating the AWS Status page.

With some lame ass tiny blue "connectivity issues" informational text. Surely broken routing to two entire DCs is full red for all services available therein?

Like what, the networking is broken but if you could send packets, the services would still work so they are green?

Re: Tell HN: AWS appears to be down again

#338
post #329

There was a brief period of time back in the early 90's where I felt I understood how Linux worked -- the kernel, startup scripts, drivers, processors, boot tools, etc... I could actually work on all levels of the system to some degree. Those days are long gone. I am far removed from many details of the systems I use today. I used to do a lot of assembly programming on multiple systems. Today I am not sure how most o…

To an extent, this is one of the goals, to free up engineers to work on higher level things. Whether it meets that goal in some cases is debatable, and it’s certainly not ideal for us engineers who like to get to the bottom of things.

“working on higher level things” currently implies that depending on many layers of opaque and unreliable lower level hardware and software abstractions is a good idea. I think it is a mistake.

Re: Tell HN: AWS appears to be down again

#339

Earlier quoted context omitted.

> Cluster berries, from raspberries to pineapples, are never poisonous. I know nothing but that still seems too generalized that it doesn’t have an exception somewhere in the world.

You won't find enough berries, never mind edibles berries if everyone in your area suddenly shifted to foraging. Game animals would be exhausted quickly or they would migrate further out from human settlements. Even if you had the skills hunting or foraging isn't all that useful anywhere around a city, especially if everyone else is doing it.

Our Hoopa lived virtually exclusively on shellfish for hundreds of years.
Post reply on HN