781–790 of 870 posts
Give people a bonus when things didn’t break, not only when there is a superhero that fixes broken things. Then you’re rewarding fragile systems that need superheroes.
Earlier quoted context omitted.
Agreed, and also it's worth noting that we're talking about companies here. Yes, for any individual the amount of money lost is insane, but that's the risk for the company. If one individual can accidentally nearly bankrupt the company, then the company did not have proper risk management in place. That isn't too say that it wouldn't also affect my sleep quality.
Sadly a lot of managers don't see it this way, they'd rather assign blame.
Earlier quoted context omitted.
Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).
Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.
Earlier quoted context omitted.
When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".
Welp, as a new grad there, I had brought down one very important database server on a Sunday night (a series of really unfortunate events). Multiple senior DBAs had to be involved to resuscitate it. It started functioning normally just a few hours before market open in HK. If it was any later, it would have been some serious monetary loss. Needless to say, I was sweating bullets. Couldn't eat anything the entire day…
Earlier quoted context omitted.
Total world economic output is ~$150M / minute, so billions every minute is off by few orders of magnitude.
You are assuming that a minute of disruption can not cause more than a minute's loss of productivity. I don't think that assumption is justified. Consider an exactly one minute outage that affects multiple things I use for work. First, I may not immediately recognize that the outage is actually with some single service provider. If several things are out I'm probably going to suspect it is something on my end, or may…
Earlier quoted context omitted.
Welp, as a new grad there, I had brought down one very important database server on a Sunday night (a series of really unfortunate events). Multiple senior DBAs had to be involved to resuscitate it. It started functioning normally just a few hours before market open in HK. If it was any later, it would have been some serious monetary loss. Needless to say, I was sweating bullets. Couldn't eat anything the entire day…
as a new grad there, it wasn't your fault. There should be guardrails to protect you.
During the outage though, no one (obviously) had time for me. This was a very important server. The tension and anxiety on the remediation call was through the roof. Every passing hour someone even more important in the chain of command was joining the call. At that time I thought I was done for...
Earlier quoted context omitted.
https://en.wikipedia.org/wiki/Billion A billion is a number with two distinct definitions: - 1,000,000,000, i.e. one thousand million, or 10^9, as defined on the short scale. This is now the meaning in both British and American English. - 1,000,000,000,000, i.e. one million million, or 10^12, as defined on the long scale. This is one thousand times larger than the short scale billion, and equivalent to the short scal…
This is a financial discussion though so: https://www.worldometers.info/gdp/gdp-by-country/ World's GDP is $80,934,771,028,340 (nominal, 2017). https://www.wolframalpha.com/input/?i=%2480%2C934%2C771%2C02... $80.93477102834 trillion Nobody would argue world GDP is anything billion, that's crazy.
In France, they use milliard and billion.
Earlier quoted context omitted.
Nope, especially considering the implications of this, with the amount of people working remotely. Google Meet, Classroom, etc. are down. This is probably literally costing billions every minute just in loss of productivity.
Total world economic output is ~$150M / minute, so billions every minute is off by few orders of magnitude.
I’d guess actual losses to the world economy were more on the order of about $100K per minute, or about 1/3 of Google’s revenue. MAYBE a few hundred thousand per minute, though that seems unlikely with Search being unaffected, and everything else coming back. Certainly a far cry from billions per minute :)
Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.
Many years ago, I was directly responsible for causing a substantial percentage of all credit/debit/EBT authorizations from every WalMart store world-wide to time out, and this went on for several days straight.
On the ground, this kind of timeout was basically a long delay at the register. Back then, most authorizations would take four or five seconds. The timeout would add more than 15 seconds to that.
In other words, I gave many tens of millions of people a pretty bad checkout experience.
This stat (authorization time) was and remains something WalMart focuses quite heavily on, in real time and historically, so it was known right away that something was wrong. Yet it took us (Network Engineering) days to figure it out. The root cause summary: I had written a program to scan (parallelized) all of the store networks for network devices. Some of the addresses scanned were broadcast and network addresses, which caused a massive amplification of return traffic which flooded the satellite networks. Info about why it took so long to discover is below.
Back in the 1990s, when this happened, all of the stores were connected to the home office via two way Hughes satellite links. This was a relatively bandwidth limited resource that was managed very carefully for obvious reasons.
I had just started and co-created the Network Management team with one other engineer. Basically prior to my arrival, there had been little systematic management of the network and network devices.
I realized that there was nothing like a robust inventory of either the networks or the routers and hubs (not switches!) that made up those networks.
We did have some notion of store numbers and what network ranges were assigned to them, but that was inaccurate in many cases.
Given that there were tens of thousands of networks ranges in question, I wrote a program creatively called 'psychoping' that would ICMP scan all of those network ranges with adjustable parallelism.
I ran it against the test store networks, talked it over with the senior engineers, and was cleared for takeoff.
Thing is, I didn't start it right away; some other things came up that I had to deal with. I ended up started it over a week after review.
Why didn't this get caught right away? Well, when timeouts started to skyrocket across the network, many engineers started working on the problem. None of the normal, typical problems were applicable. More troubling, none of the existing monitoring programs looked for ICMP at all, which is what I was using exclusively.
So of course they immediately plugged a sniffer into the network and did data captures to see what was actually going on. And nothing unusual showed up, except a lot of drops.
We're talking > 20 years ago, so know that "sniffing" wasn't the trivial thing it is now. Network Engineering had a few extremely expensive Data General hardware sniffers.
And to these expensive sniffers, the traffic I was generating was invisible.
Two things: the program I wrote to generate the traffic had a small bug and was generating very slightly invalid packets. I don't remember the details, but it had something to do with the IP header.
These packets were correct enough to route through all of the relevant networks, but incorrect enough for the Data General sniffer to not see them.
So...there was a lot of 'intense' discussions between Network Engineering and all of the relevant vendors. (Hughes, ACC for the routers, Synoptics and ODS for the hubs)
In the end, a different kind of sniffer was brought in, which was able to see the packets I was generating. I had helpfully put my userid and desk phone number in the packet data, just in case someone needed to track raw packets back to me.
Though the impact was great, and it scared me to death, there were absolutely no negative consequences. WalMart Information Systems was, in the late 1990s, a very healthy organization.
> Did they try to fix them by inverting a binary tree?
>> Yeah maybe implementing a quick LRU cache on the nearest whiteboard will help them out here
>> Did they try checking what shape their manhole cover is?
>> Dev ops was too busy out counting all the street lights in the United States
[1] https://www.reddit.com/r/programming/comments/kcwqij/every_s...