Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#221

Post by 2muchcoffeeman »

CodesInChaos wrote: post #207
maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

> some kinds of JOINs can go exponential. How? AFAIK a single join can be at most quadratic, and multiple joins should at most be polynomial, where the exponent is the number of joins. To go exponential, you'd need some kind of recursion or self reference, but I know no way to express such a thing as an ordinary join statement. (of course quadratic performance is already prohibitively slow on large tables, so there i…

Unfortunately exponential sometimes means “grows rapidly” and not exponential in a mathematical sense.

Re: Ask HN: Best “I brought down production” story?

#222

Post by r00f »

I was moving customer's two servers to virtual machines. They told me "do not touch server A, it has network share with pdf documents that are critical for our business. Take server B and make VMs there".

So I format the drive, install win server on it, and the owner runs in with red face "omg did you delete it? it was on server B!"

And of course there were no backups. And pdf files were indeed crazy important. The customer is in panic and despair, he is crying that we must remove the hard drive and take it to some data restoration company. We shut down the server, and they ask me to check accountant PC before I leave, she was complaining that it is very slow.

I turn PC on and I get pop up notification "network share is off, but your autonomous files are available". It turns out she put the checkbox on that network share by accident, but now we were sitting in front of her desktop and looking at absolutely intact folder with all pdf files. We succesfully finished moving both servers to VMs after celebrating.

That was the only time in my life when I saw that checkbox in real action.

Re: Ask HN: Best “I brought down production” story?

#224

Post by r00f »

lights0123 wrote: post #94
bentcorner wrote: post #69

Earlier quoted context omitted.

Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.

Recovery disk, then either copy the disk's copy of bash (if it doesn't depend on a later glibc version), copy another shell to /bin/bash (as the system probably doesn't depend on bash-specific commands to boot), chroot and use the package manager, or use the package manager with an explicit sysroot (e.g. pacman --sysroot). The first two steps are very easy compared to the latter two, but should be followed by a reins…

i wonder, wouldn't copy over sftp work? i don't think it depends on shell

Re: Ask HN: Best “I brought down production” story?

#225

Post by jorams »

Izkata wrote: post #216
sjaak wrote: post #157

Earlier quoted context omitted.

Medivia? Judging from screenshots that looks a lot like Tibia ( https://www.tibia.com )

Never heard of Medivia before, but looking at screenshots and their site, that is absolutely some sort of fork/copy of Tibia. Even the world map is basically same shape, with locations shifted slightly and renamed, and the game interface is the same. I think they may even be using the old sprites from before a UI revamp a bit over a decade ago, from when I played Tibia. Edit: Found this [0] which states Medivia origi…

Since they mentioned the late 90s, I think it's more likely they meant Medievia[1], which has been online since 1992.

[1]: http://www.medievia.com/

Re: Ask HN: Best “I brought down production” story?

#226

Post by mal-masri »

I just started working in an E-comerce hosting company less than a year ago with very little experince, I mean other than a C Programming course in university I had absolutely no experience so I literally had to learn everything from the beginning, starting with using Linux to Databases and some programming languages and more... but I was really Happy that it was working.

a few Months in, while working on my second sql export that i developed in a test environment which had the scheme of the companies Database, the test environment froze and then the whole Operating System of the laptop froze too. I repooted not thinking anything of it giving that my work laptop had done that a few Times already, i then proceeded to testing my export using the live system thinking i just needed more computation power, after all what's the worst that can happen !? sure enough the web interface of the Database freezes too at which point i went back to developing the export in the test environment. Some 40 Minutes later i get a question from the senior Programmer at the Time if i did anything on the live Database cause the whole thing froze up and with it no costumer service was possible for that time, thankfully he have already fixed it by repooting the Systeme and proceeded to tell me that the export that i have tried had more possibilities than the number of atoms on earth... so yeah i did need more computation power

Re: Ask HN: Best “I brought down production” story?

#229

Post by avh02 »

When i was working for one of the biggest regional websites:

We had blue/green deployments using AWS Elastic Beanstalk - our actual deployment process was still manual and we had an informal (i.e: not written down but everybody knew) checklist to follow - one of the steps involved checking that capacities matched.

Well, everything looked good and I switched environments, as I was checking that the website was still up, i was getting 503's. I realised immediately that I forgot to actually scale up the environment that was now receiving traffic - 1 server was handling what we generally had a minimum of 30 to handle.

Immediately swapped back, scaled up, and swapped again. ~5 minutes of downtime in the morning.

Re: Ask HN: Best “I brought down production” story?

#230

Post by heavenlyblue »

jfrunyon wrote: post #170
ramoz wrote: post #27

Nothing crazy, but something I always laugh at. I was so excited to meet a legit/professional dev team the first day of my career. I was paired with a Sr dev and sat in his cubicle so he could show me some backend update to a prod app with 20K internal users... "normally I'd run this on the dev server, but, its quick & easy so Ill just do it on prod" ...watched him crash the whole thing & struggle the rest of the day…

The amount of "yep, I know" I have to do any time anything goes down... at least Sales usually only has one person report it to me, other teams 3 different people will tell me at the same time...

You could send them an email the moment you know so that they wouldn’t feel like they need to report that.
Post reply on HN