Live data from Hacker News

Google Intrusion Detection Problems

fredtrotter.com

61–70 of 145 posts

Re: Google Intrusion Detection Problems

#61
post #56
post #47

Earlier quoted context omitted.

EC2 instances are meant to be ephemeral.

That doesn't sound factual at all. Got docs which support that? They send us emails every time they plan on doing intentional maintenance on our instances.

I wish it wasn't true. On all the occasions, I was told that there was "an issue with the underlying hardware".

Re: Google Intrusion Detection Problems

#62
post #34

My name is Terrance and I work on Google's Cloud Support Team. Our team mission is to Reduce Customer Anxiety. We take that very seriously, and Fred's experience obviously shows that we fell short this time. The process should have been an easy flow to follow, and it was not. We are reviewing this incident in detail to ensure that we make the process less error prone and quicker in the future. Best, Terrance

I don't want to sound disrespectful, and very much appreciate your posting. But realize that the audience here has seen the OP's story play out with Google's products over and over again. This is clearly a systemic issue with how Google handles support, and claiming "fell short this time" is more than a little disingenuous. (I also get 'man bites dog' will also get more traction than the inverse, but regardless - thi…

FWIW Google Cloud Support != Google support in general. My interactions with Cloud Support on the other side of the fence has led me to believe they are very committed and working very hard. Yes, you have to pay for it. But once you do, regardless of the tier of support, I've seen them work tirelessly and escalate tickets to engineering quickly (order hours, not days) if they couldn't figure it out.

I'm looking at the postmortem now and without wishing to jump the gun and talk about things I can't, it looks like this is being taken very seriously and a number of improvements and bug fixes are going to result. In this instance, I think it's doing the Cloud Support people a disservice to call them "disingenuous".

Disclaimer: Used to work on Google Cloud, now on Google Open Source Programs Office.

Re: Google Intrusion Detection Problems

#63
post #61
post #56

Earlier quoted context omitted.

That doesn't sound factual at all. Got docs which support that? They send us emails every time they plan on doing intentional maintenance on our instances.

I wish it wasn't true. On all the occasions, I was told that there was "an issue with the underlying hardware".

Sure, but that's very different than them being intended as ephemeral.

Re: Google Intrusion Detection Problems

#64

Of course it's easy to pattern match this to previous stories of advertisers and publishers being banned from Google's ad products, people losing their email access on Gmail, etc, but this is an entirely different area of the company, and one where account managers are very much a thing. It strikes me as odd that one couldn't just email or phone their account manager to ask for clarity on a situation like this?

Google tries to get away with as little human supervision as possible. I doubt they have an actual account manager

[deleted]

Re: Google Intrusion Detection Problems

#65

Of course it's easy to pattern match this to previous stories of advertisers and publishers being banned from Google's ad products, people losing their email access on Gmail, etc, but this is an entirely different area of the company, and one where account managers are very much a thing. It strikes me as odd that one couldn't just email or phone their account manager to ask for clarity on a situation like this?

Google tries to get away with as little human supervision as possible. I doubt they have an actual account manager

They have an account manager, it's just a robot named Samantha West, programmed to insist she's a real person:

http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarke...

Re: Google Intrusion Detection Problems

#66
post #63
post #61

Earlier quoted context omitted.

I wish it wasn't true. On all the occasions, I was told that there was "an issue with the underlying hardware".

Sure, but that's very different than them being intended as ephemeral.

I didn't suggest that those are ephemeral.

Re: Google Intrusion Detection Problems

#67

My name is Terrance and I work on Google's Cloud Support Team. Our team mission is to Reduce Customer Anxiety. We take that very seriously, and Fred's experience obviously shows that we fell short this time. The process should have been an easy flow to follow, and it was not. We are reviewing this incident in detail to ensure that we make the process less error prone and quicker in the future. Best, Terrance

It seems like there are a lot of points in Fred's story where even a miniscule amount of manual testing would have revealed the problem. For example, paying a tester to follow the instructions in the FAQ or to try to appropriately respond to the original email would have revealed how unclear the instructions were and that there is a missing UI control.

Re: Google Intrusion Detection Problems

#68

Earlier quoted context omitted.

> Is Amazon or Azure or Heroku really any different? Nope, and thats why I don't use them. There are situations where they make sense, but people should be much more skeptical about them. I don't have stats, but IME I've yet to see a company that actually saves money by using them. I've seen one company that did save money the first year, but got too addicted to "just spinning another instance up" and stopped optimiz…

I worked on a project that costs about $5k/month to self-host. After moving it to AWS, it was about $2k/month. So $36k/year savings in hosting costs. In our case outgoing bandwidth at AWS was about 10x cheaper than what our ISP offered. Having spent years help running a physical hosting company, working with virtualized servers saves LOTS of time. I still find it amazing and much, much cheaper in staff-hours to reboo…

That funny, because one of my beefs with Amazon is that they charge a lot for bandwidth! Once you're in a co-lo, high quality bandwidth just isn't that expensive.

Re: Google Intrusion Detection Problems

#69

Of course it's easy to pattern match this to previous stories of advertisers and publishers being banned from Google's ad products, people losing their email access on Gmail, etc, but this is an entirely different area of the company, and one where account managers are very much a thing. It strikes me as odd that one couldn't just email or phone their account manager to ask for clarity on a situation like this?

It's part of Google culture, Google DNA. Even though the departments are different, the heritage will show.

Re: Google Intrusion Detection Problems

#70
post #61
post #56

Earlier quoted context omitted.

That doesn't sound factual at all. Got docs which support that? They send us emails every time they plan on doing intentional maintenance on our instances.

I wish it wasn't true. On all the occasions, I was told that there was "an issue with the underlying hardware".

Yes, this happens sometimes with AWS. We're heavy users and have had to respawn instances on several different occasions due to unexpected issues with the underlying hardware. Only once or twice has AWS told us they're moving us because of that, but we've noticed performance issues that after troubleshooting with support, could only be described as a symptom of the hardware.

Everyone should have backups and a plan in place to recover when a node goes offline, but this isn't exclusive to AWS or cloud in general. A robust infrastructure will have failovers standing by and accept the possibility that any one machine could die at any moment. Shouldn't have any single points of failure, as colocated hardware kicks the bucket unexpectedly sometimes too.

Post reply on HN