Live data from Hacker News

Google outage – resolved

news.ycombinator.com

701–710 of 870 posts

Re: Google outage – resolved

#701
post #682

Earlier quoted context omitted.

No Engineer should have production access from their workstation. Period. source: am Engineer =).

Why not? (I think I can find some cases where production access from an engineer's workstation is a good idea)

It can be efficient, particularly in smaller companies, but that's where exactly this rule should be applied.

In some industries, security and customer requirements will at times mandate that developer workstations have no access to production. Deployments must even be carried out using different accounts than those used to access internal services, for security and auditing purposes.

There are of course good reasons for this; accidents, malicious engineers, overzealous engineers, lost/stolen equipment, risk avoidance, etc.

When you apply this rule, it makes for more process and perhaps slower response times to problems, but accidents or other internal-related issues mentioned above drop to zero.

Given how easy it is to destroy things these days with a single misplaced Kubernetes or Docker command, safeguards need to be put in place.

Let me tell you a little story from my experience;

I built myself a custom keyboard from a Numpad kit. I had gotten tired of typing so many docker commands in every day and I had the desire to build something. I built this little numpad into a full blown Docker control centre using QMK. A single key-press could deploy or destroy entire systems.

One day, something slid off of something else on my desk, onto said keyboard, pressing several of the keys while I happened to have an SSH session to a remote server in focus.

Suffice it to say, that little keyboard has never been seen since. On an unrelated topic, I don't have SSH access to production systems.

Re: Google outage – resolved

#702
post #369

Earlier quoted context omitted.

Some politics topics have shot up pretty quickly in the past but user and/or mod flags send them back to page 2 about as quickly.

Funny this post is already #7. Are we seeing many reports again? Stats right now: 1985 points | 1 hour ago | 597 comments

Maybe, could have just seen a dropoff in upvotes once the issue was resolved also

Re: Google outage – resolved

#703
post #646

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".

I'm reminded of the quote from Thomas J. Watson:

> Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?

Re: Google outage – resolved

#705

Gmail said my account was "temporarily" unavailable... had a moment considering if it wasn't temporary. Good reminder to remove my reliance on gmail especially.

Our entire library of video tutorials disappeared for a while. I was not happy, and the thought of losing our email.. Currently working with the team to make backups of absolutely everything off-site and off-Google. A good wake up call for us.

Re: Google outage – resolved

#706

Earlier quoted context omitted.

No Engineer should have production access from their workstation. Period. source: am Engineer =).

Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.

Not having those things centralized is also a huge operational failure regardless of company size.

Re: Google outage – resolved

#707

Earlier quoted context omitted.

No Engineer should have production access from their workstation. Period. source: am Engineer =).

Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.

For a large or v large tech co you should probably be aggregating logs to a centralised location that doesn't require access to production systems in this way. Stack dumps should also be collected safely off-system if necessary.

Perhaps my industry is a little more security conscious (I don't know which industry you're talking about), but this doesn't seem like good practice.

Re: Google outage – resolved

#708

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

I am not sure why are they allowing it. Meaning why aren’t services completely isolated? Isn’t it obvious that in an intertwined environment those things are bound to happen (as in “question of when, not if”)? I understand, in smaller companies that are limited in resources (access to good developers and pressure to get product to market as soon as possible) we have single points of failure all over the place. But “the smartest developers on the planet”? What is it if not short-sighted disregard for risk management theories and practices? I mean, Calendar and Youtube, say, should be completely separate services hosted in different places, their teams should not even talk to each other. Yes, they can use same software components, frameworks and technologies. Standardization is very welcome. But decentralization should be an imperative.

Edit: again downvotes started! Thanks to everyone “supporting freedom of expression” :)

Re: Google outage – resolved

#709
post #638

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

I remember how one of our engineers had his docker daemon connected to production instead of his local one and casually did a docker rm -f $(docker ps -aq) . Same thing happened to me but with CI, which felt bad enough already.

"Hey let's make developers do two very different jobs, development, and operations. We'll call it DevOps. We'll save money. Everything will be fine."

Re: Google outage – resolved

#710
post #642

Earlier quoted context omitted.

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.

colorblind (red/green) person here - 5% of the male population just don't see color enough for it to be an important visual clue.

So sure, color-code your environments, but if you find someone about to do something to a red environment that they clearly should only be doing to a green environment, just check if they're seeing what you're seeing before you sack them ;)

Post reply on HN