Google Cloud Is Down • Hacker News Board

Re: Google Cloud Is Down

#301

Post by the-rc » Sun, Jun 02, 2019, 9:38 PM UTC

boulos wrote: post #151

boulos wrote: post #63

Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!). We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages. There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns do…

To clarify something: this outage doesn’t appear to be global, but it is hitting us particularly hard in parts of the US. So for the folks with working VMs in Mumbai, you’re not crazy. But for everyone with sadness in us-central1, the team is on it.

Fiber cut? SDN bug that causes traffic to be misdirected? One or more core routers swallowing or corrupting packets?

Re: Google Cloud Is Down

#302

Post by thruhiker » Sun, Jun 02, 2019, 9:38 PM UTC

ljm wrote: post #219

murat124 wrote: post #167

Earlier quoted context omitted.

You obviously don't recall the early years of AWS. Half of internet would go down for hours.

Always in Virginia, because US-east has always been cheaper.

The only regions that are more expensive than us-east-1 in the States are GovCloud and us-west-1 (Bay Area). Both us-west-2 (Oregon) and us-east-2 (Ohio) are priced the same as us-east-1.

Re: Google Cloud Is Down

#303

Post by dodobirdlord » Sun, Jun 02, 2019, 9:39 PM UTC

vpontis wrote: post #42

mandatory wrote: post #18

Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.

Praying isn't working. Now, I'll try sobbing :(

Reports claim that it's a network congestion issue primarily in the northeast United States. It seems doubtful that any data has been lost, but you probably can't see resources due to requests not getting through. I hope that hearing this helps you to feel better.

Re: Google Cloud Is Down

#304

Post by SmokeGS » Sun, Jun 02, 2019, 9:39 PM UTC

stanfordkid wrote: post #153

SmokeGS wrote: post #82

Earlier quoted context omitted.

>nothing is lost except time

Can't use my Nest lock to let guests into my house. I'm pretty sure their infrastructure is hosted in Google Cloud. So yeah... definitely some stuff lost.

holy shit lmao. I'm sorry that sucks.

Re: Google Cloud Is Down

#305

Post by disillusioned » Sun, Jun 02, 2019, 9:39 PM UTC

stanfordkid wrote: post #153

SmokeGS wrote: post #82

Earlier quoted context omitted.

>nothing is lost except time

Can't use my Nest lock to let guests into my house. I'm pretty sure their infrastructure is hosted in Google Cloud. So yeah... definitely some stuff lost.

Sure you can, but you'll need to give them your code or the master code. Unless you've enabled Privacy Mode, in which case... I don't know if even the master code would work.

Re: Google Cloud Is Down

#306

Post by marcinzm » Sun, Jun 02, 2019, 9:40 PM UTC

WC3w6pXxgGd wrote: post #107

w_s_l wrote: post #83

You know this reminds me of a bad taste that Google Sales team left when I asked for some of my billing that I was unaware of running after following a quickstart guide. AWS refunded me in the first reply on the same day! GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me. So why not just tell your customers like it…

Anything created in-house at Google (GCP) is typically created by technically-proficient devs, those devs then leave the project to start something new and maintenance is left to interns and new hires. Google customer service basically doesn't care and also has no tools at their disposal to fix any issues anyway. The infinite money spout that is Google Ads has created a situation in which devs are at Google just to h…

Isn't it also that promotions at Google are based on creating new products/projects rather than maintaining existing ones? So engineers have a negative incentive to maintain things since it costs them promotions.

Re: Google Cloud Is Down

#307

Post by twistedpair » Sun, Jun 02, 2019, 9:41 PM UTC

gundmc wrote: post #290

tntn wrote: post #269

There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount for the month? https://cloud.google.com/compute/sla

A couple more hours and everyone will get 25% off for June.

Does that apply to the rest of June?

Might be a good month to rebuild all your models ;)

Re: Google Cloud Is Down

#308

Post by squarefoot » Sun, Jun 02, 2019, 9:41 PM UTC

I appears that logging into the webmail solves at least POP mail problems. I tried my mail client and failed, then attempted a login to the webmail which worked. Gmail then asked me to confirm my recovery address and cellphone, which I did, and finally loaded the inbox page. I immediately attempted a connection through the POP client and this time it worked.

It might be something security related if it triggers a mandatory identity confirmation.

edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error.

[23:44:27] POPThe problem seems much more complex.

Re: Google Cloud Is Down

#309

Post by pg_is_a_butt » Sun, Jun 02, 2019, 9:41 PM UTC

None of my stuff is down.

Re: Google Cloud Is Down

#310

Post by dodobirdlord » Sun, Jun 02, 2019, 9:42 PM UTC

ramon wrote: post #273

It seems that it`s focused on the database stuff like firebase and firestore.

Apparently (from further up the thread) it's a network congestion issue causing extremely high rates of packet loss. I imagine pretty much anything that's homed in the effected regions will be degraded or unaccessible.