Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident report for February 21st, 2024

resend.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Incident report for February 21st, 2024

#41

Post by JackFr »

sergiotapia wrote: post #34
yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

I did this once early in my career, the ice cold feeling in the pit of my stomach permanently etched this lesson into my soul lol.

I did too, but it was 25 years ago and quite honestly there was a different attitude about developers accessing production.

Today if a developer can bring down the operation accidentally, that’s a problem with the org more than the developer.

(On the other hand if a developer screws up the shared dev environment, it is his or her fault and they deserve the wrath of their coworkers.)

Re: Incident report for February 21st, 2024

#42

Post by namaria »

pistoriusp wrote: post #29

Unfortunately these sort of mistakes are seen as a "right of passage" for many developers. I ran "`DELETE FROM users;` without a WHERE clause against production in my first year on the job. I felt absolutely terrible. I thought I was connected to a development machine. Fortunately we had backups available. Often this isn't a problem with the individual developer itself, but points to a problem with the organization.…

I think you mean 'rite of passage'

Re: Incident report for February 21st, 2024

#43

Post by doctor_eval »

pistoriusp wrote: post #30
doctor_eval wrote: post #17

If they’re small then you can see it happening where someone was logged into prod using some environment variables to sort out some issue - probably didn’t even update production, just a few queries - and then went back to work. Hours later they run some script that does DROP DATABASE from that same shell they used to troubleshoot, which takes a little longer than usual… Anyway I can totally see it happening to me in…

Or just use https://www.snaplet.dev

Unclear how that would have helped.

Re: Incident report for February 21st, 2024

#44

Post by tetha »

tedchs wrote: post #36

I did this, around 2005, but I dropped ALL the prod tables. I was using a SQL GUI called Toad (awesome) and had separate windows open, for both "dev" and "prod". I was trying to reset the dev database, and used the wrong window. Thankfully, the "real" DBA at the time had 15-minute backups, and was able to restore it, and then I replayed a few transactions from logs. Lesson learned! > While building a feature, we perf…

This is why I've conditioned myself: Once I start running unsafe commands in a development-ish pattern (DDL changes, reboots, service starts/stops and such), I make sure to close any shell to productive systems and databases first. They are color-coded differently, normal users don't have these permissions and so on and so on, sure.

But it's better not to have these shells open and available to accidentally choose them once my brain switches from ops-mode to dev-mode.

Re: Incident report for February 21st, 2024

#45

Post by dbbk »

laborcontract wrote: post #19

Man, I use Resend, I really want to like them and yeah, it is really simple to get started which is great but man, this is, I think they really need to slow down a bit and maybe try to figure out how to put some processes in place to maybe intentionally slow things down. This is the second incident that can be characterized by a very hazy delineation between development and production environments. The first incident…

I don't say this to be rude, but honestly it's because they are an amateur company. When you're dealing with critical infrastructure like email, you should stick to the battle-tested services like SES, Postmark, Sendgrid etc.

Re: Incident report for February 21st, 2024

#46

Post by dbbk »

albertgoeswoof wrote: post #27
walrus01 wrote: post #25

Earlier quoted context omitted.

They do, or do not operate MX and deliver SMTP traffic to other mail servers?

They are a wrapper over AWS SES

Wait really?? So what is it you're actually getting for a presumably marked-up price?

Re: Incident report for February 21st, 2024

#47

Post by dpcx »

I did something like this nearly 15 years ago at a job. I was trying to use something like MySQL Workbench to export the database and generate a map of connections through foreign keys. Apparently I selected some option backwards and it deleted out the entire production database.

Luckily for me I'd been working on some other things related to it and had taken a backup not long prior, but it was pretty nerve wracking to hear the CTO/CEO nearly running through the halls to find out what had happened.

Pretty sure they never implemented more stringent access controls at that company, either.

Re: Incident report for February 21st, 2024

#48

Post by viiralvx »

anonzzzies wrote: post #11
disillusioned wrote: post #8

Earlier quoted context omitted.

That single sentence contains multitudes: * Production should be immutable * No one doing dev in a dev environment should have such trivial access to prod * Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state? Yikes.

> No one doing dev in a dev environment should have such trivial access to prod It’s the new and hip ‘cloud’! Probably using planetscale or something like that, which (last I checked, maybe it changed but wasn’t on), doesn’t even have ip protections outside the mysql user settings (while bad, would’ve protected them). > Are there still good reasons for a migration to drop all tables? We haven’t found any.

I'd suggest reading up on what some of these new database providers are doing to help prevent or fix mistakes like this. Since you mentioned PlanetScale, I'll use them as an example.

1) PlanetScale has IP ACLs, which locks down passwords to specific IP addresses. [1] Additionally, with TailScale or another VPN solution, locking down based on IP isn't necessary foolproof.

2) They also have Safe Migrations. When enabled, it prevents DDL from being run directly on a database. [2] Additionally, using deploy requests for zero-downtime schema migrations also allows you to use reverts, which will revert the migration. [3]

[1] https://planetscale.com/blog/introducing-ip-restrictions

[2] https://planetscale.com/docs/concepts/safe-migrations

[3] https://planetscale.com/blog/behind-the-scenes-how-schema-re...

Re: Incident report for February 21st, 2024

#49

Post by samlambert »

rwieruch wrote: post #26
anonzzzies wrote: post #11

Earlier quoted context omitted.

> No one doing dev in a dev environment should have such trivial access to prod It’s the new and hip ‘cloud’! Probably using planetscale or something like that, which (last I checked, maybe it changed but wasn’t on), doesn’t even have ip protections outside the mysql user settings (while bad, would’ve protected them). > Are there still good reasons for a migration to drop all tables? We haven’t found any.

PlanetScale has Safe Migrations which you can enabled for your production DB (branch). Wondering though whether this will protect against everything mentioned here. https://planetscale.com/docs/concepts/safe-migrations

Safe Migrations would prevent this completely. PlanetScale also allows you to restore multiple backups in parallel.

Re: Incident report for February 21st, 2024

#50

Post by seanwilson »

inopinatus wrote: post #12
syslog wrote: post #10

Earlier quoted context omitted.

How does immutable work in regard to databases? As in „we need to add a column“?

Someone writes the migration, commits it, it passes the build and unit test stages of the pipeline, then the application as currently running passes all function and integration tests with (and this is important) both the prior and the revised schema. Your commit is tagged as release ready! Not long after, the automation tooling confidently executes the now-tested migration under machine control during the next deplo…

> Someone writes the migration, commits it, it passes the build and unit test stages of the pipeline, then the application as currently running passes all function and integration tests with (and this is important) both the prior and the revised schema. Your commit is tagged as release ready! Not long after, the automation tooling confidently executes the now-tested migration under machine control during the next deploy, everyone goes home happy

What happens if something goes really wrong after the production deploy? Is there a way to skip steps if you need to quickly push an emergency fix?

Post reply on HN