The day I locked everyone out of the company intranet

Frequently heard in batcaves: "Hey, the FNG børked Prod."

"One of us!! One of us!!" :-)

Roughly 10 years ago, I was working for a startup that offered a live conversational video service where you could also have hundreds (or eventually, thousands) of near-live watchers - with recording and later playback. The founder pitched the service to news orgs and celebrities. Anderson Cooper had a regular "show" there for a while, and we had a number of interviews with mostly 2nd-tier celebrities.

When the service started, they made the decision to not actually delete any content (delete just set a flag which disabled the content but didn't actually remove it).

Fast forward a year or so, and it became clear that a real delete was needed. So they had a junior engineer write up a sort of delayed sweep - delete all the videos with the delete flag set. But then, for some reason, they decided put the implementation behind a delay. Something like "actually delete all soft-deleted videos, but don't start doing it until 30 days from now". However, unbeknownst to the team, there was a bug in the implementation that deleted everything, regardless of whether the 'delete' flag was set.

So one night, roughly a month later, all the content started disappearing from the site. One guy heroically tried to stop the process, but I think he was too late. The engineering director happened to be on a vacation down in South America somewhere and I think the founder fired him in a fit of pique. I managed to reclaim a small bit of content (basically the videos that were cached on the actual recording servers before they were uploaded to S3).

You can imagine the technical over-reaction:

  * Delete switched back to a soft delete
  * Turned on S3 object versioning
  * Started redundantly copying content onto a totally different hosting service

This was fine (hah!) until we had to start taking down the inevitable child porn that always shows up on services like this - I got stuck with writing the takedown code and it took me forever to track down all the various tendrils of stuff.

As you might expect, we lost a ton of users over mass content deletion and the service never really rebounded. The company held on for a couple more years, pivoting a couple of times, but eventually folded.

Autocommit being off took prod down for me.

Hop in the test database, run some SQL, update a bazillion records, run some selects to verify, open the test app and verify there. All good! Someone else checks… LGTM.

So I do the same thing in prod. Make changes, validate in the DB, looks good. Open the prod app… down. Requests are hanging.

Our software is shit so I’m not surprised. I start frantically digging through trying to figure out where and why it’s hanging. Eventually chase it down to some lock wait timeouts in SQL. Start looking at the database… no queries seem to be running against the table in question. Eventually dig a little deeper and find… the call was coming from inside the house! The locks are held by my connection. Go hit “commit” in dbeaver and prod comes right back up.

And that’s the boring story of the day I learned that dbeaver used transactions in prod connections by default.

FTA - Make friends with your sysadmin, and stay in their good books at all costs.

I think this goes against every principle most on HN have.

A lot of older systems were like that before they became more commodity items with guard rails.

Some of the early SAN disk arrays used to use an assembly like config interface with no sanity checks, type in the wrong command and you could wipe the whole array.

The company I worked for at the time had a blanket policy that any change had to be implemented by the vendor to shift liability for any mistakes.