Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident report for February 21st, 2024

resend.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Incident report for February 21st, 2024

#31

Post by tianzhou »

disillusioned wrote: post #8
yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

That single sentence contains multitudes: * Production should be immutable * No one doing dev in a dev environment should have such trivial access to prod * Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state? Yikes.

We have been working on bytebase (https://github.com/bytebase/bytebase) for 3+ years to address this. With a change review workflow, environment propagations, and try not to disturb the dev flow if possible.

Re: Incident report for February 21st, 2024

#32

Post by rnts08 »

This will keep happening as long as people are unable to learn from the past. Yes it's expensive to have a good experienced infrastructure engineer on the team, but at least you know there's someone testing your backups and procedures for when your eager dev team screw up.

Re: Incident report for February 21st, 2024

#33

Post by tianzhou »

pistoriusp wrote: post #29

Unfortunately these sort of mistakes are seen as a "right of passage" for many developers. I ran "`DELETE FROM users;` without a WHERE clause against production in my first year on the job. I felt absolutely terrible. I thought I was connected to a development machine. Fortunately we had backups available. Often this isn't a problem with the individual developer itself, but points to a problem with the organization.…

We are also building Bytebase, which enforces the change review process for such operations

Re: Incident report for February 21st, 2024

#34

Post by sergiotapia »

yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

I did this once early in my career, the ice cold feeling in the pit of my stomach permanently etched this lesson into my soul lol.

Re: Incident report for February 21st, 2024

#35

Post by ZephyrBlu »

ttul wrote: post #5
yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

This implies that their production environment is mutable. As in, a command can run and change the production environment. That’s a no no. But I give them a pass because they are a young company. My company was similarly reckless early on, but as we scaled, we had to tighten things up and turning to an immutable deployment approach has saved our asses so many times.

Is being a young company really an excuse when any half-decent engineer knows these things are bad?

Being a young company doesn't mean you ignore all the mistakes other people have made and figure them out for yourself.

I really surprised someone has access to the prod DB, and that it's possible for them to connect to it in dev (Meaning they have a copy of the credentials???).

Re: Incident report for February 21st, 2024

#36

Post by tedchs »

I did this, around 2005, but I dropped ALL the prod tables. I was using a SQL GUI called Toad (awesome) and had separate windows open, for both "dev" and "prod". I was trying to reset the dev database, and used the wrong window. Thankfully, the "real" DBA at the time had 15-minute backups, and was able to restore it, and then I replayed a few transactions from logs. Lesson learned!

> While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production.

Re: Incident report for February 21st, 2024

#38

Post by walrus01 »

albertgoeswoof wrote: post #27
walrus01 wrote: post #25

Earlier quoted context omitted.

They do, or do not operate MX and deliver SMTP traffic to other mail servers?

They are a wrapper over AWS SES

so, great, something that still serves the purpose of funneling more unsolicited marketing email (spam) into peoples' inboxes, just with a shiny wrapper on it.

Re: Incident report for February 21st, 2024

#39

Post by JackFr »

tedchs wrote: post #36

I did this, around 2005, but I dropped ALL the prod tables. I was using a SQL GUI called Toad (awesome) and had separate windows open, for both "dev" and "prod". I was trying to reset the dev database, and used the wrong window. Thankfully, the "real" DBA at the time had 15-minute backups, and was able to restore it, and then I replayed a few transactions from logs. Lesson learned! > While building a feature, we perf…

I deleted a years worth of trades on a production database for a bond trading system in 1997. They were restored within an hour, but it made a lasting impression.

Different color scheme on prod consoles.

Always use “BEGIN TRAN”.

Re: Incident report for February 21st, 2024

#40

Post by JackFr »

yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

At a company with a modern, mature process this simply should not be possible.
Post reply on HN