41–50 of 52 posts
Never run prod in us-east-1.
I would argue the opposite. Run in us-east-1 if you want to have truly reliable infrastructure.
> Existing instances and networks continue to work normally. This suggests that one can minimize certain failure modes by making the configuration as static as possible, i.e. creating a fixed number of long-running instances. But then, using immutable, ephemeral instances can make for a more resilient system when the EC2 control plane is working normally. There are always tradeoffs.
There actually are several good reasons to run in us-east-1. Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almo…
Earlier quoted context omitted.
Downtime is kinda hard to grok for huge distributed services like this. If you’re down when anyone gets errors you’ll probably always be down — not really useful. If you’re down when everyone gets errors you’ll probably always be up — again not really useful. From a customer perspective an increase in errors means nothing if it doesn’t affect you or everything if it does so even a flat percentage is likely not all th…
Hence why they people use percentile measurements for alerting at places like AMZN. This person worked on Alexa and wrote an article about it: https://medium.com/@djsmith42/how-to-metric-edafaf959fc7
But if you measure something as a percentile, and it's just my "shard" or whatever, I'm like 0.0001% of S3 traffic, so you'll never notice it at that scale. A small constant stream of errors is somewhat acceptable, so long as they're just the randomly distributed noise that is random failures and bit flips. But persistent errors that block forward progress or some customer having 0% success is what causes SWEs to ask themselves "I wonder what GCP is like.", and it becomes very clear that their monitoring doesn't indicate when the service is on fire for someone, only when it is on fire for everyone.
Azure had an outage yesterday (that was only visible to customers / wasn't reflected on the public status page): there's apparently no more capacity in West US for certain VMs. And, well, it is sort of a basic supposition that our cloud provider can provide a cloud, when asked. And region hopping is not simple: a whole host of resources just aren't compatible if you're not in the same region. (E.g., a vnet requires the VMs be in the same region.) (And that was "elevated" error rates, or something, again. Where, as a customer, "elevated" would appear to mean "100%".) Not a great start to the week.
Someone posted Rachel-by-the-Bay's "Your nines are not my nines" article in a sibling comment, which is basically this. People way discount the TCO of the cloud, IMO.
I'd also add I've worked at companies where every 5xx was hunted down. It was easy to get complacent, chalk them up to "oh, probably just a network blip" and we thankfully had a staff eng. who was very good and log diving and showing that no, it wasn't a blip, it was a bug in our code, and we should fix that bug.
There actually are several good reasons to run in us-east-1. Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almo…
Earlier quoted context omitted.
Nobody uses us-east-1 anymore. It's too crowded.
All the new instance types seem to get deployed on US-EAST-1 first. It may be crowded, but it always gets the good stuff first.
There actually are several good reasons to run in us-east-1. Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almo…
I run and own a DevOps consulting company, and NEVER provision AWS resources in us-east-1. If a east coast location is need, I use us-east-2 (Ohio).
There actually are several good reasons to run in us-east-1. Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almo…