Earlier quoted context omitted.
Use `date -u -Iseconds`, please ;-)
date: illegal option -- I usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
AWS us-east-1 down
221–230 of 331 posts
Re: AWS us-east-1 down
#222For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.
Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.
Nothing worse than people who say "9 AM my time" I suppose it's OK if it's Pacific vs Mountain but even there Arizona doesn't observe Daylight, and parts of Eastern Oregon are Mountain, not Pacific.
Never mind dealing with India, Australia, etc etc.
OK to use local time in your statement, just say what that time is.
Re: AWS us-east-1 down
#223Earlier quoted context omitted.
1) and 2) certainly apply. 3) and 4) don't. Testing in the largest region is one of the biggest anti-patterns.
4 is still generally true. Most new features drop in us-east-1 on launch day.
Usually us-east-1 is deployed to after several smaller regions. Usually it'll fall in the middle of the week depending on the pipeline.
Just because a feature is there on launch day doesn't mean it was deployed to first. Features are often hidden behind flags that are switched for launch.
Re: AWS us-east-1 down
#224Why is it always us-east-1 though? I have always stayed away from that region because it seems significantly less reliable than other regions.
I thought I read that this is where they deploy new changes first. Can anyone confirm?
When I worked there, there were few hard and fast rules. Every team had its own release processes, so there was a lot of variance. It has been a couple of years, so this may have changed.
Typically, a team would group their regions into batches and deploy their change to one batch at a time. Usually they follow a geometric progression, so the first batch has one region, the second batch has two regions, the third batch has four regions, and so on. This batching was performed for the sake of time; nobody wants to wait a month for a single change to finish rolling out.
One reason not to deploy to us-east-1 in the first batch is so you don't blow up your biggest region. The fewer customers you break, the better.
One reason not to deploy to us-east-1 in the last batch is that there are a lot of batches. If a problem is uncovered after deploying the last batch, then someone has to initiate rollbacks for every single region.
Some teams tried to compromise and put us-east-1 in one of the earlier batches.
Re: AWS us-east-1 down
#225I posted a comment about AWS Media Convert down below, but it's back working for me.
Re: AWS us-east-1 down
#226Earlier quoted context omitted.
4 is still generally true. Most new features drop in us-east-1 on launch day.
Usually us-east-1 is deployed to after several smaller regions. Usually it'll fall in the middle of the week depending on the pipeline. Just because a feature is there on launch day doesn't mean it was deployed to first. Features are often hidden behind flags that are switched for launch.
I'm well aware of that, but the point is that when the feature is ungated to the public, it's in us-east-1 and gets all that load, and more load than the rest because of the fact that a lot of big customers are based in us-east-1, including much of Amazon itself.
Re: AWS us-east-1 down
#227Earlier quoted context omitted.
that's based on UTC, so just use UTC?
based on is not the same thing though is it? UTC is human readable even if it is not calculated correctly. yes, i'm saying that if you can read epoch seconds, you're not human. 1970-01-01 00:00:00 is always a give away that something is a foot
"anchored on" then? I might be wrong but we're both talking about showing time as distance from the same starting point are we not? One's just more human readable so that's why I say why not just use that? Seconds since can be miscalculated too, especially if current time isn't known/reliable
Re: AWS us-east-1 down
#228Why does everyone keep deploying their products to this one region when it always seems like the one that fails? We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?
It's dirt cheap.
But I still prefer EU region =).
Re: AWS us-east-1 down
#229Why is it always us-east-1 though? I have always stayed away from that region because it seems significantly less reliable than other regions.
I thought I read that this is where they deploy new changes first. Can anyone confirm?
No definitely not. Usually pipelines deploy over 1-2 week periods, and they don't deploy on Fridays/holidays/high-traffic periods like December.
Deployments start off very conservative, maybe 1-2 small regions on the first day of deployments. As you gain confidence, the pipeline deploys to more regions/bigger regions.
A pipeline that deploys to 22 regions over one week might go from 2 small regions on monday, 4 small/medium regions on tuesday, 8 medium/large regions on wednesday, 8 regions on thursday.
us-east-1 is usually going to be deployed to on the wednesday/thursday in this example, but that isn't always the case because sometimes deployments are accelerated for feature launches (especially around re:invent), or retried because of a failure.
There are best practice guides within Amazon that very closely detail how you should deploy, although it is up to the teams to follow them, which they usually do an okay job of.
Re: AWS us-east-1 down
#230Why does everyone keep deploying their products to this one region when it always seems like the one that fails? We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?
Features.
For example, do you want your Cloudfront CDN to have a custom (secure) domain?
Then you have to host your ACM cert in us-east-1.