Now you see that it's not only Microsoft problem...
Google's Down
51–60 of 67 posts
Re: Google's Down
#52Discord was down for several minutes earlier, as well. Related?
Re: Google's Down
#53Spain went down for 5 min while i was in a google doc + analytics. I run downforeveryoneorjustme.com and we went up to 1000 people a minute checking too. It only lasted a few min though.
I just want to say thank you for your website. It’s saved my sanity many times. I’m curious, what’s your tech stack? Your site always seems to be up when everyone else is down. :)
Thanks :)
My friend spent the last year rewriting it! She is amazing! We went from Ruby on Rails, AWS, and a $800+ a month server bill to paying almost nothing to run it (and with better checking).
We are working on launching a blog for it and she is going to write up a full breakdown of the project. I'll post it to Show HN then.
This is a write up that covers it a bit from how we got featured on CloudFlare last week: " As for how we're using Cloudflare workers...
The actual frontend website is a fairly tiny Nuxt/VueJS app and that operates in AWS Lambda right now, but we have about an 80% cache rate normally, so Cloudflare is serving most of that anyway. It's on AWS primarily because we have credits with them, plus that is where I started working on it and I had used Lambda in previous projects. At the time we relaunched it early last year, Cloudflare workers was still in beta and not as refined as it is now, especially now that it has better CLI tooling. The only reason the site itself doesn't live in Workers is because deploying JS apps was a bit...rough last year due to lack of tooling but it looks like that has improved significantly and I will probably eventually move the frontend site, too.
The backend portion of the site that checks if a site is down is just a simple HTTP API, which also was first written for Lambda/Node, but I later rewrote it in Cloudflare Workers primarily because I liked that it executed the worker in the edge closest to the visitor automatically, instead of us having to deploy our app in numerous AWS regions. It's a perfect use case for how our site works.
The other way we use Workers is in conjunction with the key/val store service you have. Our site gets a lot of attacks and I made a rate-limiting system that will rate limit and then automatically block abusers that do not relent in the cloudflare firewall via your API. It uses ELK + Elastalert to accomplish that, which gives us a bit more control over how and when we want to rate limit / block than the one in built-in to Cloudflare. I struggled a bit getting these under control with AWS (even with their WAF) and had a fairly frequent events that paged me requiring manual intervention, but now I barely ever get paged for issues on the site, it's been 100% uptime for months.
I'm a huge proponent of serverless if the project fit is right and I've definitely enjoyed working with Cloudflare a lot more than Lambda. "
Re: Google's Down
#54Re: Google's Down
#55Earlier quoted context omitted.
There are many warnings now.
Seeing that now. I like how the outage times are back-timed (11:57am est) to well before they posted anything about it.
That seems reasonable to me. You could argue that the posting should be quicker, but it's never going to be instantaneous. Back-timing is at least somewhat transparent.
Re: Google's Down
#56Meanwhile their status page[0] is all green. The less they publish on their status page the more nines they preserve. [0] https://status.cloud.google.com/ Edit: status page not entirely green now, took them 25 minutes to acknowledge.
G Suite Status is all green with no warnings. https://www.google.com/appsstatus
Good to know that Google+ is healthy
Re: Google's Down
#57Authoritative explanation
Very sorry about that! We had a router failure in Atlanta, which affected traffic routed through that region. Things should be back to normal now. Just to make sure: this wasn't related to traffic levels or any kind of overload, our network is not stressed by Covid-19.
Re: Google's Down
#58Looks like issues are primarily on the east coast: https://downdetector.com/status/google/map/ No issues with any google services thus far for me in Denver.
Im in Denver too. consumer products are up.
Cloud: we have large appEngine (2000 req/s at peak - it's very up and down spiking), functions, and heavy pubsub/biquery. Didn't notice any downtime or large errors.
Last time there was cloud downtime Google wouldn't give any credit because it was still under the SLA. Though we're really small fish a tiny gesture of good will would have felt nice.
Re: Google's Down
#59Authoritative explanation Very sorry about that! We had a router failure in Atlanta, which affected traffic routed through that region. Things should be back to normal now. Just to make sure: this wasn't related to traffic levels or any kind of overload, our network is not stressed by Covid-19. https://twitter.com/uhoelzle/status/1243217659690278912
Thank you. Upvote this!
Re: Google's Down
#60For posterity, the final status update:
The issue with Google Cloud infrastructure components has been resolved for all affected users as of 09:21. Total time of impact was 08:18 to 09:21 US/Pacific, with the most severe impact at the start of the issue, tapering off as services routed traffic away from Atlanta.
The impact of this incident was concentrated in a region that is not a main GCP region and therefore there was no impact to services based on Google Compute Engine. Services that may have been impacted include External HTTP/S Load balancing requests and API requests that may have been served near the Atlanta metro.
The root cause was a set of router failures in Atlanta, which affected traffic routed through that region.