Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident report for February 21st, 2024

resend.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Incident report for February 21st, 2024

#11

Post by anonzzzies »

disillusioned wrote: post #8
yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

That single sentence contains multitudes: * Production should be immutable * No one doing dev in a dev environment should have such trivial access to prod * Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state? Yikes.

> No one doing dev in a dev environment should have such trivial access to prod

It’s the new and hip ‘cloud’! Probably using planetscale or something like that, which (last I checked, maybe it changed but wasn’t on), doesn’t even have ip protections outside the mysql user settings (while bad, would’ve protected them).

> Are there still good reasons for a migration to drop all tables?

We haven’t found any.

Re: Incident report for February 21st, 2024

#12

Post by inopinatus »

syslog wrote: post #10
ttul wrote: post #5

Earlier quoted context omitted.

This implies that their production environment is mutable. As in, a command can run and change the production environment. That’s a no no. But I give them a pass because they are a young company. My company was similarly reckless early on, but as we scaled, we had to tighten things up and turning to an immutable deployment approach has saved our asses so many times.

How does immutable work in regard to databases? As in „we need to add a column“?

Someone writes the migration, commits it, it passes the build and unit test stages of the pipeline, then the application as currently running passes all function and integration tests with (and this is important) both the prior and the revised schema. Your commit is tagged as release ready! Not long after, the automation tooling confidently executes the now-tested migration under machine control during the next deploy, everyone goes home happy with your shiny new published_at column, and no-one has directly touched prod.

Two days later the CTO sends everyone a stroppy email about "column bloat that should've been a table", ssh's into the personal instance that they've been keeping alive† since before you had funding and learned to launch servers as immutable black boxes, and whilst trying to prove a point by rolling it back manually, drops all tables by mistake when a cat treads on the keyboard

--

† excuse: "it's for reporting"

Re: Incident report for February 21st, 2024

#14

Post by kgeist »

Reminds of this classic thread: https://www.reddit.com/r/cscareerquestions/s/cqrama0L1z

>accidentally destroyed production database on first day of job

Same symptoms: while developing a feature locally, they accidentally pointed to the production DB, which was destroyed when running tests.

Re: Incident report for February 21st, 2024

#15

Post by kgeist »

satvikpendem wrote: post #7
anonzzzies wrote: post #6

This sounds like one of those horrible tools like prisma which drop everything if something is not in sync on dev. We removed this type of stupid in favour of our own which, you know, fixes this actually instead of lazily dropping everything when they cannot resolve some trivial thing, for instance, a new required field without default when there are already rows and other crap which they call ‘opinionated’. No idea…

I like that Prisma drops the database. Developer databases should be idempotent and easily reseedable with sample data. It's the same concept as cattle, not pets, from the devops world but now applied to databases. There should be nothing special about a database in development that cannot be dropped and recreated.

We have a script which recreates the dev DB, but developers must run the command manually so that there was an understanding what's happening.

Re: Incident report for February 21st, 2024

#16

Post by willsmith72 »

anonzzzies wrote: post #11
disillusioned wrote: post #8

Earlier quoted context omitted.

That single sentence contains multitudes: * Production should be immutable * No one doing dev in a dev environment should have such trivial access to prod * Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state? Yikes.

> No one doing dev in a dev environment should have such trivial access to prod It’s the new and hip ‘cloud’! Probably using planetscale or something like that, which (last I checked, maybe it changed but wasn’t on), doesn’t even have ip protections outside the mysql user settings (while bad, would’ve protected them). > Are there still good reasons for a migration to drop all tables? We haven’t found any.

really? i've definitely done it before on my local as a quicker alternative to cleaning up the docker container/volume, doesn't seem that bad

ofc i'd think differently if i was also putting write-permission prod credentials into my machine, but luckily i haven't been in many places doing that

Re: Incident report for February 21st, 2024

#17

Post by doctor_eval »

If they’re small then you can see it happening where someone was logged into prod using some environment variables to sort out some issue - probably didn’t even update production, just a few queries - and then went back to work.

Hours later they run some script that does DROP DATABASE from that same shell they used to troubleshoot, which takes a little longer than usual…

Anyway I can totally see it happening to me in my little one man shop, but now i think I might look at removing some privileges from my prod account :)

Re: Incident report for February 21st, 2024

#18

Post by RowanH »

inopinatus wrote: post #12
syslog wrote: post #10

Earlier quoted context omitted.

How does immutable work in regard to databases? As in „we need to add a column“?

Someone writes the migration, commits it, it passes the build and unit test stages of the pipeline, then the application as currently running passes all function and integration tests with (and this is important) both the prior and the revised schema. Your commit is tagged as release ready! Not long after, the automation tooling confidently executes the now-tested migration under machine control during the next deplo…

Rookie move having the cat on the desk while ssh'd into prod...

Re: Incident report for February 21st, 2024

#19

Post by laborcontract »

Man, I use Resend, I really want to like them and yeah, it is really simple to get started which is great but man, this is, I think they really need to slow down a bit and maybe try to figure out how to put some processes in place to maybe intentionally slow things down.

This is the second incident that can be characterized by a very hazy delineation between development and production environments. The first incident had to do with an attacker gaining access to private credentials due to devs leaving keys set in NEXT_PUBLIC environmental variables on their site.

Re: Incident report for February 21st, 2024

#20

Post by vmfunction »

walrus01 wrote: post #13

From the company's homepage: "deliver marketing emails at scale" Maybe this company doesn't need to exist, and shouldn't.

It is actually very open of them to post that! I'm sure things like that happens in larger companies, but I'm sure we won't get a incident report like that from them. I would rather work with that kind of company. Then we can also see if they learned from their lesson, and this doesn't repeat again.
Post reply on HN