Google Cloud networking issues in us-east1

Not sure why they closed that one at 9:12 just to open a new one at 10:25. We didn't see any traffic coming to us-east1 during that time period so I would assume the original issue is still the root cause.

Re: Google Cloud networking issues in us-east1

#5

Post by zbowling » Tue, Jul 02, 2019, 7:10 PM UTC

thsowers wrote: post #3

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/item?id=20213092 [1]: https://news.ycombinator.com/item?id=20077421

unrelated. very big company with thousands of products that don't suffer outages. two incidents doesn't make a pattern.

Re: Google Cloud networking issues in us-east1

#6

Post by inlined » Tue, Jul 02, 2019, 7:11 PM UTC

Holy crap. It’s an outage in all zones? What’s the point of AZs if you lose whole DCs at a time.

Re: Google Cloud networking issues in us-east1

#7

Post by thsowers » Tue, Jul 02, 2019, 7:13 PM UTC

zbowling wrote: post #5

thsowers wrote: post #3

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/item?id=20213092 [1]: https://news.ycombinator.com/item?id=20077421

unrelated. very big company with thousands of products that don't suffer outages. two incidents doesn't make a pattern.

I would argue that two direct Google Cloud outages within a month is pretty concerning for GCP customers, and that it's possible that the calendar outage could also be related in someway since it is likely hosted on GCP, although that is speculation

Re: Google Cloud networking issues in us-east1

#8

Post by klodolph » Tue, Jul 02, 2019, 7:13 PM UTC

inlined wrote: post #6

Holy crap. It’s an outage in all zones? What’s the point of AZs if you lose whole DCs at a time.

Availability is hierarchical.

Re: Google Cloud networking issues in us-east1

#9

Post by estsauver » Tue, Jul 02, 2019, 7:13 PM UTC

In a moment that's likely to be very, very frustrating for a large number of you that have businesses and customers that depend on G cloud, let's try to remember that somewhere there's an engineer or an SRE having a really hard day just trying to fix things.

Please, be kind and decent to each other, especially when things are hard.

Re: Google Cloud networking issues in us-east1

#10

Post by joshuamorton » Tue, Jul 02, 2019, 7:14 PM UTC

fastest963 wrote: post #4

Here's the original issue: https://status.cloud.google.com/incident/cloud-networking/19... Not sure why they closed that one at 9:12 just to open a new one at 10:25. We didn't see any traffic coming to us-east1 during that time period so I would assume the original issue is still the root cause.

Hopefully the thread title can be updated. (If it were actually down, this thread would have been posted 3 hours ago and have 400+ comments).