Incident report for February 21st, 2024
resend.com
Incident report for February 21st, 2024
1–10 of 65 posts
Re: Incident report for February 21st, 2024
#2"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production."
This was scary.
Re: Incident report for February 21st, 2024
#3"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.
Separation between office and production should be the norm.
Re: Incident report for February 21st, 2024
#4"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.
Extremely so in my opinion. In this day and age, viewing something like this means this company fundamentally doesn't understand security. It should simply be impossible to "incorrectly point to the production environment", because whoever ran this shouldn't even have access to those credentials in the first place.
Re: Incident report for February 21st, 2024
#5"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.
This implies that their production environment is mutable. As in, a command can run and change the production environment. That’s a no no.
But I give them a pass because they are a young company. My company was similarly reckless early on, but as we scaled, we had to tighten things up and turning to an immutable deployment approach has saved our asses so many times.
Re: Incident report for February 21st, 2024
#6This sounds like one of those horrible tools like prisma which drop everything if something is not in sync on dev. We removed this type of stupid in favour of our own which, you know, fixes this actually instead of lazily dropping everything when they cannot resolve some trivial thing, for instance, a new required field without default when there are already rows and other crap which they call ‘opinionated’. No idea why we ever used that stuff as it caused so much grief (never on prod though); after prisma/drizzle and some other ‘modern’ horrors, we mistrust everything that’s ’hip and new’ so ‘everyone uses it’. One of those hip and new things I mistrusted was Resend even though ‘suddenly everyone uses it’. I’ll wait a few years before even contemplating it.
Re: Incident report for February 21st, 2024
#7This sounds like one of those horrible tools like prisma which drop everything if something is not in sync on dev. We removed this type of stupid in favour of our own which, you know, fixes this actually instead of lazily dropping everything when they cannot resolve some trivial thing, for instance, a new required field without default when there are already rows and other crap which they call ‘opinionated’. No idea…
I like that Prisma drops the database. Developer databases should be idempotent and easily reseedable with sample data. It's the same concept as cattle, not pets, from the devops world but now applied to databases. There should be nothing special about a database in development that cannot be dropped and recreated.
Re: Incident report for February 21st, 2024
#8"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.
That single sentence contains multitudes:
* Production should be immutable
* No one doing dev in a dev environment should have such trivial access to prod
* Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state?
Yikes.
Re: Incident report for February 21st, 2024
#9This sounds like one of those horrible tools like prisma which drop everything if something is not in sync on dev. We removed this type of stupid in favour of our own which, you know, fixes this actually instead of lazily dropping everything when they cannot resolve some trivial thing, for instance, a new required field without default when there are already rows and other crap which they call ‘opinionated’. No idea…
I like that Prisma drops the database. Developer databases should be idempotent and easily reseedable with sample data. It's the same concept as cattle, not pets, from the devops world but now applied to databases. There should be nothing special about a database in development that cannot be dropped and recreated.
I don’t care about dev databases; I care about why it does it; there is no need for it, at all.
> I like that Prisma drops the database.
‘Like’ ; so if it didn’t but still worked perfectly, you would complain that it didn’t?
Re: Incident report for February 21st, 2024
#10"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.
This implies that their production environment is mutable. As in, a command can run and change the production environment. That’s a no no. But I give them a pass because they are a young company. My company was similarly reckless early on, but as we scaled, we had to tighten things up and turning to an immutable deployment approach has saved our asses so many times.
How does immutable work in regard to databases? As in „we need to add a column“?