Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

171–180 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#171

Post by jiggawatts »

mtsr wrote: post #155
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

And this is why it’s not easy to use any of the cloud platforms parttime. They’re all full of these kinds of gotchas and your example is even a minor one since it really is quite clear from the documentation. There’s even a choice of Provisioned Throughput if you’re clicking this together in the Console, which should probably make you suspicious about what happens if you don’t use that. But a certified AWS Architect,…

What value is the certification for the cloud? It's such an incredibly rapidly changing thing, anything you learn goes out-of-date in a matter of months. The two things I complained about in this thread are no longer issues, they got fixed.

The project I'm working on right now is using a PaaS stack is a bit of a "moving target". I feel like those people running to catch a train that's still rolling along...

Re: Post Mortem of Google Outage on 14 December 2020

#172

Post by jiggawatts »

b112 wrote: post #150
jiggawatts wrote: post #127

Earlier quoted context omitted.

Ahhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course. The golden rule of both backups and monitoring is: There are no time machines. It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame…

I'll be the fly in the ointment here, the additional sand in your shorts, and say "This is why bare metal is better". Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS. And yes, that's with the hardware investment and wage costs rolled in. I fee…

Hardware has the exact same monitoring issues, worse even.

The dinosaurs of the on-prem hardware world like Dell, HPE and IBM make the most atrocious systems management software that I have ever seen. Bargain basement quality at best.

The cloud is eating their lunch for a reason.

Everyone wants a single pane of glass, not a hundred unique and special vendor-specific consoles to manage one app.

Everyone wants unified logging built-in, not Splunk on top of fifty different log formats, none of which can be easily correlated with each other.

Everyone wants a unified IAM system, nobody likes to deal with expired SAML or LDAP certificates in the middle of a change.

Etc, etc...

Compared to having to deal with 10 different teams just to spin up a single VM and 15+ teams for a moderately complex application with HA/DR and monitoring, being able to simply click through a cloud portal is an absolute joy.

It's the same thing that made VMware so popular. Nobody liked to have to get finance involved and order new kit six weeks ahead just to be able to spin up a tiny web server.

Re: Post Mortem of Google Outage on 14 December 2020

#173

Post by BugWatch »

herf wrote: post #95

In Thunderbird, OAuth2 login is still broken. The login page prompts for email again and again, never makes it to the password.

I am having the same difficulties, and it's giving me a headache. The only workaround was to change the authentication method to "Normal password", and enable the "Allow less secure apps" setting on the Gmail account.

And that's a major PITA since I have 40-ish Gmail accounts. (Why so many? I tend to separate my different concerns for privacy and security reasons.)

Re: Post Mortem of Google Outage on 14 December 2020

#175

Post by BugWatch »

aleph1 wrote: post #148
herf wrote: post #95

In Thunderbird, OAuth2 login is still broken. The login page prompts for email again and again, never makes it to the password.

This could be caused by Google not recognizing and blocking Thunderbird's default user agent; try toggling general.useragent.compatMode.firefox to true (which basically has TB emulate Firefox's user agent)

Thank you! This worked in my case.

Re: Post Mortem of Google Outage on 14 December 2020

#176

Post by halflings »

Angostura wrote: post #174

I'm not an infrastructure engineer - could someone explain the benefits of using quotas for something like an authentication service. It feels like something that shouldn't really need a quota - unless the idea is to monitor services that have run amok.

Any service running way over quota could break all other services. (at the end of the day, resources are physically limited to what you have in your datacenters)

Quotas are one way to isolate this impact to that service in particular.

Of course, when it's a critical service like authentication, it hardly isolates anything... but I can't think of a better alternative.

Re: Post Mortem of Google Outage on 14 December 2020

#177

Post by dijit »

frankchn wrote: post #136
silentsea90 wrote: post #118

Earlier quoted context omitted.

S3 is strongly consistent. https://aws.amazon.com/s3/consistency/ Which of Google's nosql db provides strong consistency - bigtable? Just confirming

S3 became strongly consistent only recently ( https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea... ) while I think GCS and Azure Blob Storage has strong read-after-write consistency for a while now. In any case, Cloud Spanner provides strong consistency in multi-region deployments.

And GCP storage buckets has been built on top of spanner sine 2018- giving the same guarantees.

If anything. AWS is playing catch up.

Re: Post Mortem of Google Outage on 14 December 2020

#178

Post by aljarry »

ineedasername wrote: post #108
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…

Around 2 years ago in S3 there was an interesting quote issue - official documentation stated quotas per S3 prefix (kind of like folder path), but in practice it turned out only first 20+ characters of a prefix were taken into account for quotas, so our burst computations loading thousands of objects were mostly timing out. We had to add random prefixes at the beginning just to get around that. AWS updated their quota management to per-full-prefix, finally though.

Re: Post Mortem of Google Outage on 14 December 2020

#180

Post by HeavyStorm »

jugg1es wrote: post #126
ineedasername wrote: post #108

Earlier quoted context omitted.

Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…

Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue i…

Wow, that's a scary story.
Post reply on HN