441–450 of 630 posts
And thus was ruined hundreds or thousands of pleasant Sunday afternoons. I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.
The only response is to wait for Google to fix it. Nothing you or I or the pager can do will speed that up. I am aware some bosses won't believe that and I am not trying to make light of it. But there really isn't much else to do except wait.
Earlier quoted context omitted.
Care to name names? I've been looking for a small, cheap failover for a moderately low traffic app.
In the US I use Hivelocity. If you want cheapest possible, Hetzner/OVH have deals you can get for _cheap._
Earlier quoted context omitted.
It seems global to me. This is really strange compared to AWS. I don't remember an outage there (other than s3) impacting instances or networking globally.
Where are you based? If you’re in the US (or route through the US) and trying to reach our APIs (like storage.googleapis.com), you’ll be having a hard time. Perhaps even if the service you’re trying to reach is say a VM in Mumbai.
Earlier quoted context omitted.
Any cloud provider offering those terms would go out of business VERY quickly. Outages happen, all providers are incentivized to minimize the frequency and severity of disruptions - not just from the financial hit of breaching SLA (which for something like this will be significant), but for the reputational damage which can be even more impactful.
> Any cloud provider offering those terms would go out of business VERY quickly Minimum spends and a 50,000% markup based on adding that term to your contract.
Earlier quoted context omitted.
Just like AWS, then. "Some users are experiencing increased error rates" = "Everything has been down for hours"
I remember when S3 was down and the status was green because the updates for the status page with pushed via S3.
Earlier quoted context omitted.
The only response is to wait for Google to fix it. Nothing you or I or the pager can do will speed that up. I am aware some bosses won't believe that and I am not trying to make light of it. But there really isn't much else to do except wait.
Or you wait for Google or you are frantically trying to move everything you got to AWS.
If you try to be heroic, you get back to 100% with a bunch of wasted effort and stress on your part.
Because it will be fixed by Google, regardless of what you do or don't do.
After the incident is over would be the time to consider alternatives.
Earlier quoted context omitted.
Everyone talking about security and not replacing locks with smart locks seems to forget that you can just kick the fucking door down or jimmy a window open.
Or just sawzall a hole in the side of the house...
Earlier quoted context omitted.
I am not sure if a single S3 outage pushed any big names into their own "datacenter". S3 has still the world record of reliability that you cannot challenge with your inhouse solutions. You can prove it otherwise. I would love to hear a solution that has the same durability, avabiality and scalability as S3. For the downvoters, please just link here the proof if you disagree. Here are the S3 numbers: https://aws.amaz…
It's not so much AWS vs. in-house. But AWS (or GCP/DO/etc.) vs. multi/hybrid solutions. The latter of which would presumably have lower downtime.
Earlier quoted context omitted.
Google is well known for not caring about small shops, only if you are a multi million dollar customer with dedicated account manager you can expect reasonable support. That's been the case forever with them.
Does Amazon treat smaller customers any better? I am genuinely asking, as I have no clue.
Earlier quoted context omitted.
There was a massive push after that to have everything regionalized. It's not 100% but it's super close at this point.
S3 buckets are a global namespace, so control plane operations have to be single-homed. As an example, global consensus has to be reached before returning a success response for bucket creation to ensure that two buckets can't be created with the same name.