Google Cloud Is Down • Hacker News Board

Re: Google Cloud Is Down

#611

Post by namibj » Mon, Jun 03, 2019, 12:13 PM UTC

juanuys wrote: post #586

based2 wrote: post #266

https://www.reddit.com/r/sysadmin/comments/bw1gye/most_googl... https://www.zdnet.com/article/some-internet-outages-predicte... 768k Day

Will this affect more than just Google? I haven't seen any outages from other cloud providers.

packet.net was hit. Specifically, also their San Jose DC. Internet only. It took less than an hour to recover. More than 20 minutes. I didn't ping it continuously, and I can say that the traceroute got stuck in Frankfuhrt (Where my ISP and their ISP first (as seen from me) meet).

I was actually surprised, as they tend to have excellent networking. Now I'm not nearly as distrusting as I was initially, knowing it was likely their ISP getting screwed by routing table overflow.

Re: Google Cloud Is Down

#612

Post by andrewjshults » Mon, Jun 03, 2019, 12:44 PM UTC

curiouscats wrote: post #231

gowld wrote: post #185

Earlier quoted context omitted.

It's a trade-off of risks. Leaving a key under the may could lead to a security breach.

I am pretty sure there are smart locks that don't rely on an active connection to the cloud. The lock downloads keys when it has a connection and a smartphone can download keys. This means they work even if no active internet connection at the time the person tries to open. If the connection was dead the entire time between creating the new key and the person trying to use the lock it still wouldn't work. If there ar…

FWIW - The Nest lock in question doesn't rely on an active internet connection to work. If it can't connect, it can still be unlocked using the sets of PINs you can setup for individual users (including setting start/end times and time of day that the codes are active). There's even a set of 9V battery terminals at the bottom in case you forget to change the batteries that power the lock.

This does mean you need to setup a code in advance of people showing up, but it's an under 30 second setup that I've found simpler than unlocking once someone shows up. The cameras dropping offline are a hot mess though, since those have no local storage option.

Re: Google Cloud Is Down

#613

Post by chupasaurus » Mon, Jun 03, 2019, 1:00 PM UTC

dijit wrote: post #546

chupasaurus wrote: post #207

Earlier quoted context omitted.

Any solution would be based on third parties, the robust solution is either to run your own country with fuel sources for electricity and army to defend the datacenters or rely on multiple independent infrastructures. I think the latter is less complex.

This is a ridiculous statement. Surely you realise that there is a sliding scale. You can run your own hardware and pull in multiple power lines without establishing your own country. I’ve ran my own hardware, maybe people have genuinely forgotten what it’s like, and granted, it takes preparation and planning and it’s harder than clicking “go” in a dashboard. But it’s not the same as establishing a country and source…

The GP's statement is about relying on third parties, multiple power lines with generators you don't own on the other side falls under it.

Fun related fact: My first employee's main office was in former electonics factory in Moscow's downtown powered by 2 thermal power stations (and no other alternatives), which have exact same maintenance schedule.

Re: Google Cloud Is Down

#614

Post by aasssdddd » Mon, Jun 03, 2019, 1:03 PM UTC

w

Re: Google Cloud Is Down

#615

Post by londons_explore » Mon, Jun 03, 2019, 4:22 PM UTC

dillonmckay wrote: post #602

londons_explore wrote: post #577

Earlier quoted context omitted.

For any Google service with billing, always set up an entirely new account, in incognito mode, without using the same recovery phone number or email address. That way Google won't ban your main account for non-payment. It's the only way, especially considering Google cloud has no functionality to cap spending.

They will link the accounts by IP (and other fingerprinting), and you will get banned.

My experience shows they do not.

For a ban, they need something concrete like using the same browser cookies, recovery email address or phone number.

IP isn't enough alone - you could be on shared WiFi.

Also, after account creation, you can log in from the same place without risk, or even use multi-login to log into both accounts at the same time.

Re: Google Cloud Is Down

#616

Post by amenod » Mon, Jun 03, 2019, 5:12 PM UTC

dsumenkovic wrote: post #593

amenod wrote: post #33

Earlier quoted context omitted.

Slow is understatement... some pages on gitlab.com take minutes to load, and jobs take tens of minutes to start. EDIT: It's been like that since at least 12h ago though. Not sure if it's connected to Google Cloud?

Yes, it was definitely related to GCP services. GitLab is no longer seeing errors and Google Cloud has resolved the issue as of 23:00 UTC yesterday. Any further information can be found on the issue at https://gitlab.com/gitlab-com/gl-infra/production/issues/862

The timeline seems a bit short, but ok. :)

(the problems with runners and the UI started at least at 2019-06-02 7:48 UTC, though they were hit-and-miss at the time)

Still, happy this is solved and we can use the (fantastic otherwise) service again!

Re: Google Cloud Is Down

#617

Post by fnord123 » Mon, Jun 03, 2019, 5:40 PM UTC

fnord123 wrote: post #595

azimuth11 wrote: post #535

Earlier quoted context omitted.

I think some Google engineers published a free Meap book on service relatability and uptime guarantees. Seemingly counterintuitive, scheduling downtime, without other teams’ prior knowledge, encourages teams to handle outages properly and reduce single points of failure, among other things.

Service Reliability Engineering is on OReilly press. It's a good book. Up there with ZeroMQ and Data Intensive Applications as maybe the best three books from OReilly in the past ten years.

Derp, Site Reliability Engineering.

https://landing.google.com/sre/books/

Re: Google Cloud Is Down

#618

Post by aaronblohowiak » Mon, Jun 03, 2019, 9:32 PM UTC

justinator wrote: post #549

aaronblohowiak wrote: post #527

Earlier quoted context omitted.

We don’t usually discuss the frequency of unplanned failovers, but I will tell you that we do a planned failover at least every two weeks. The team also uses traffic shaping to perform whole system load tests with production traffic, which happens quarterly.

Do you do any chaos testing? Seems like it would slot right in, there.

https://www.oreilly.com/library/view/chaos-engineering/97814... ;)

Re: Google Cloud Is Down

#619

Post by foota » Mon, Jun 03, 2019, 10:00 PM UTC

ikiris wrote: post #570

boulos wrote: post #63

Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!). We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages. There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns do…

go/stopleaks :)

Hm, isn't releasing go links publicly also verboten? :)

Re: Google Cloud Is Down

#620

Post by thruhiker » Mon, Jun 03, 2019, 10:10 PM UTC

angstrom wrote: post #496

thruhiker wrote: post #302

Earlier quoted context omitted.

The only regions that are more expensive than us-east-1 in the States are GovCloud and us-west-1 (Bay Area). Both us-west-2 (Oregon) and us-east-2 (Ohio) are priced the same as us-east-1.

I would probably go with US-EAST-2 just because it's isolated from anything except perhaps a freak Tornado and better situated on the eastern US. Latency to/from there should be near optimal for most eastern US/Canada population.

One caveat with us-east-2 is that it appears to get new features after us-east-1 and us-west-2. You can view the service support by region here: https://aws.amazon.com/about-aws/global-infrastructure/regio....