Google Cloud networking issues in us-east1 • Hacker News Board

Re: Google Cloud networking issues in us-east1

#261

Post by m0zg » Wed, Jul 03, 2019, 2:11 AM UTC

dodobirdlord wrote: post #237

m0zg wrote: post #215

Earlier quoted context omitted.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

> 99.99% of all these outages are due to screwing up something that already works Fiber optic cables are a great technology, but they don't react well to being cut in half by a backhoe. Is the solution you are recommending that we stop using fiber optic cables, or that we stop using backhoes?

Stopping depending so much on remote datacenters unnecessarily would be a good start.

Re: Google Cloud networking issues in us-east1

#262

Post by m0zg » Wed, Jul 03, 2019, 2:15 AM UTC

owenmarshall wrote: post #235

m0zg wrote: post #221

Earlier quoted context omitted.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for d…

The first part of your post sounds like a success story to me. You got many years of use out of that server. 5 figures is cheap comparing to the cost (including the collateral damage cost) of a hotshot SWE or devops guy who insists on using the most resume-worthy, most bleeding-edge technology available.

Re: Google Cloud networking issues in us-east1

#263

Post by Thaxll » Wed, Jul 03, 2019, 2:19 AM UTC

dx87 wrote: post #77

Kind of related to this, but these types of outages are why I moved from Google Play to Spotify for streaming music. Their infrastructure seems so large that things that should be a standalone service, like streaming music, are bound to be collateral damage when they mess something up on another service. Having everything provided by one company is convenient until it all goes down at the same time and you can't acce…

I'm a heavy user of Google Play for the last 6 years, I never had a single outage since I use the service ( multiple hours a day, 5 days a week )

Re: Google Cloud networking issues in us-east1

#264

Post by akhilcacharya » Wed, Jul 03, 2019, 2:29 AM UTC

laluser wrote: post #257

akhilcacharya wrote: post #251

Earlier quoted context omitted.

I’m 90% sure it’s just to flex, honestly. I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!

Another FANG = Amazon? If so, Amazon is pretty restrictive in how it wants employees to communicate about internal activities. Most people err on the side of caution and don't comment publicly.

It is - but all companies I’ve ever worked for are. I’m not convinced the letter of the policy is much different.

Re: Google Cloud networking issues in us-east1

#265

Post by joemag » Wed, Jul 03, 2019, 2:30 AM UTC

MisterPea wrote: post #249

hunter2_ wrote: post #223

Earlier quoted context omitted.

$ host news.ycombinator.com news.ycombinator.com has address 209.216.230.240 https://whois.arin.net/rest/net/NET-209-216-230-0-1/pft?s=20... M5 Computer Security https://www.m5hosting.com Unrelated: https://begthequestion.info/

The begs the question site is one of my pet peeves. Language is not moderated by a select few who want to claim it, this isn't France. This is why Ebonics is still a valid form of English - as long as it is used consistently. If everyone uses "begs the question" and everyone else understands it as "raises the question" then it is perfectly valid.

Any phrase that succeeds at transferring an idea out of your head into mine is good enough for me.

Re: Google Cloud networking issues in us-east1

#266

Post by [deleted] » Wed, Jul 03, 2019, 2:38 AM UTC

bardworx wrote: post #213

boulos wrote: post #202

Earlier quoted context omitted.

As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opin…

That’s...that’s some petty fucking shit. I didn’t go through your comments but considering your email is in your profile, someone really had to have a hard-on to report you for leaks. I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a…

[deleted]

Re: Google Cloud networking issues in us-east1

#267

Post by 8bitchemistry » Wed, Jul 03, 2019, 2:38 AM UTC

m0zg wrote: post #221

Operyl wrote: post #218

Earlier quoted context omitted.

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

> You can't create "technical debt" if you don't change anything in the first place.

Tell that to your security team.

Re: Google Cloud networking issues in us-east1

#268

Post by praneshp » Wed, Jul 03, 2019, 2:39 AM UTC

bardworx wrote: post #213

boulos wrote: post #202

Earlier quoted context omitted.

As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opin…

That’s...that’s some petty fucking shit. I didn’t go through your comments but considering your email is in your profile, someone really had to have a hard-on to report you for leaks. I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a…

> I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN

This is not so difficult though. You just need to adjust your starting point to someone who doesn't like boulos' first. That's not so difficult IMO, it's a large org and boulos' seems to be a fairly prolific commenter here.

Re: Google Cloud networking issues in us-east1

#269

Post by boulos » Wed, Jul 03, 2019, 2:49 AM UTC

edwintorok wrote: post #203

boulos wrote: post #165

Earlier quoted context omitted.

This part was somewhat nuanced, so I wasn’t sure to post it: yes, if you are using GCLB, and have more than 1 healthy Region, we will also rebalance to avoid us-east- for now (though not so statically as that sounds, mumble mumble). Edit: added this to the top level comment so more folks see it.

There were reports of 404 from Google Cloud Run earlier today (I can confirm that I got both a 404 and a successful load after retrying that website): https://news.ycombinator.com/item?id=20336102 Was it related, it is a bit odd to get a 404 instead of a 50x?

Sorry, I hadn't seen your post earlier. No, the Cloud Run (intermittent) 404s were unrelated.

Re: Google Cloud networking issues in us-east1

#270

Post by [deleted] » Wed, Jul 03, 2019, 2:50 AM UTC

harshreality wrote: post #143

Hacker News: The real status page and help desk for the internet. Do companies realize how absurd this is? ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.

[deleted]