731–740 of 870 posts
Earlier quoted context omitted.
No Engineer should have production access from their workstation. Period. source: am Engineer =).
Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.
So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
Earlier quoted context omitted.
Why not? (I think I can find some cases where production access from an engineer's workstation is a good idea)
It can be efficient, particularly in smaller companies, but that's where exactly this rule should be applied. In some industries, security and customer requirements will at times mandate that developer workstations have no access to production. Deployments must even be carried out using different accounts than those used to access internal services, for security and auditing purposes. There are of course good reasons…
Earlier quoted context omitted.
why?
This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…
Not sure that exists for businesses, but I'd expect you'd need to go shopping separately if you want that.
Seems like a good business idea if it doesn't exist.
Earlier quoted context omitted.
This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…
Completely agree with your analogy but have you ever seen any SLA that provides any additional liability? I haven't seen them - you are stuck either relying on hosting services SLA or DIY.
Earlier quoted context omitted.
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).
So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
Unless I'm misunderstanding something, they are going to have an immense SLA claim issue. 99.9% SLA on Workspace services, so any business paying for Google for Business (now known as Workspace) is going to have a credit claim (assuming the outage is longer than 43m 49s which feels like it will be). Edit: As I comment it looks like things are coming back! Timing or what...