171–180 of 197 posts
Half the internet is down because of a Google Cloud global issue on their load balancers, including Spotify and Etsy and GCP status is all green: https://status.cloud.google.com If you ever wondered why GCP is a distant third runner in the enterprise cloud space, here is your answer.
back up as of now (10:10a PDT) This sure makes it easy to know who is hosted by google by going to downdetector.com.
Not exactly hard to do that in the first place whois $(dig +short spotify.com A) NetName: GOOGLE-CLOUD
EDIT: I was wrong, as pointed out by shizcakes.
Earlier quoted context omitted.
or https://ipfs.io/
Cool, how do I stream the newest Taylor Swift album on that?
That server continues to work just fine.
Earlier quoted context omitted.
Not exactly hard to do that in the first place whois $(dig +short spotify.com A) NetName: GOOGLE-CLOUD
That only says spotify.com is using GCP's DNS but not necessarily for hosting? EDIT: I was wrong, as pointed out by shizcakes.
Etsy is 404 too: https://www.etsy.com Seems to be a bigger issue. edit: Nest is down too: http://nest.com Fitbit.com is 404 too: https://www.fitbit.com Big GCP issue? edit2: Downdetector.com shows multiple website and services as down, including Pokemon GO or Rocket League. GCP status page is still green all over the board: https://status.cloud.google.com 19:10 CET update: Some websites are coming back, including spo…
Looks like perhaps an issue with Google Load Balancer. We have a load balancer in front of Google Storage Buckets, and can access resources directly from the buckets, but getting 404 when going through the load balancer.
Non-engineer here - Is there an easy way to multi-provider redundancy around this? Can you have LBs on multiple clouds and use dns to move around or something? Or does your LB have to be at the provider the app is at? Sorry if this makes no sense. :o
Earlier quoted context omitted.
Expecting an instant public post is a bit unrealistic. They had a post up just over 20 minutes after the incident start, which is not that crazy, given that they needed time to triage all of the alarms and understand which component was actually breaking and confirm some technically correct information around it, even if the actual internal incident response can run without the public post.
It's Google though.. they stop automating everything? At least make the screen not show all green or something automatically
If the automation is working the services will be up. When an incident is happening it's because something is significantly broken, and automation won't properly understand what is and is not working.
For instance, lots of follow-on alarms might be firing for what are not actually issues with the things being monitored: As an example, I would imagine that datacenter temperatures and fan speeds dropped due to the incident, which might cause automation to suspect a facilities issue, but announcing a facilities issue would be misleading.
Or metrics around instances live might be tanking as autoscaling groups start downsizing. This would not be an issue with the autoscaling service, and automatically announcing an autoscaling outage would again be misleading.
In an incident, taking the available data and reaching a conclusion about what is broken and what are effects is something which requires skilled manual effort and is error-prone.
Earlier quoted context omitted.
Not exactly hard to do that in the first place whois $(dig +short spotify.com A) NetName: GOOGLE-CLOUD
That only says spotify.com is using GCP's DNS but not necessarily for hosting? EDIT: I was wrong, as pointed out by shizcakes.
spotify.com. 172800 IN NS dns1.p07.nsone.net.
spotify.com. 172800 IN NS dns2.p07.nsone.net.
spotify.com. 172800 IN NS dns3.p07.nsone.net.
spotify.com. 172800 IN NS dns4.p07.nsone.net.
spotify.com. 172800 IN NS ns-cloud-a1.googledomains.com.
spotify.com. 172800 IN NS ns-cloud-a2.googledomains.com.
spotify.com. 172800 IN NS ns-cloud-a3.googledomains.com.
spotify.com. 172800 IN NS ns-cloud-a4.googledomains.com.