Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident report for February 21st, 2024

resend.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Incident report for February 21st, 2024

#51

Post by AYBABTME »

ZephyrBlu wrote: post #35
ttul wrote: post #5

Earlier quoted context omitted.

This implies that their production environment is mutable. As in, a command can run and change the production environment. That’s a no no. But I give them a pass because they are a young company. My company was similarly reckless early on, but as we scaled, we had to tighten things up and turning to an immutable deployment approach has saved our asses so many times.

Is being a young company really an excuse when any half-decent engineer knows these things are bad? Being a young company doesn't mean you ignore all the mistakes other people have made and figure them out for yourself. I really surprised someone has access to the prod DB, and that it's possible for them to connect to it in dev (Meaning they have a copy of the credentials???).

Knowing it's bad and punting on it for later are both things that can be possible at the same time.

Re: Incident report for February 21st, 2024

#52

Post by briffle »

disillusioned wrote: post #8
yatish27 wrote: post #2

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production." This was scary.

That single sentence contains multitudes: * Production should be immutable * No one doing dev in a dev environment should have such trivial access to prod * Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state? Yikes.

In addition, Every centralized storage, and every cloud provider lets you take a snapshot of a database's disks.

We can restore a 10TB disk in about 12 minutes. its much faster to snapshot, do migration, then if necessary, drop disk and remake from snapshot. (and then replay the replay any other WAL changes up to the exact second you want with a tool like barman, wall-e, pg_backreset, etc.

Postgresql backups are critical for disaster recovery, but the restores are so very, very slow, they should be a last resort.

Re: Incident report for February 21st, 2024

#53

Post by janpio »

anonzzzies wrote: post #6

This sounds like one of those horrible tools like prisma which drop everything if something is not in sync on dev. We removed this type of stupid in favour of our own which, you know, fixes this actually instead of lazily dropping everything when they cannot resolve some trivial thing, for instance, a new required field without default when there are already rows and other crap which they call ‘opinionated’. No idea…

Quick note that Prisma (and most of the other modern tools) never "drops everything" for normal migrations. A database reset is a solution suggested when your migration history and the actual database are not in sync, which makes it impossible for Prisma to "calculate" the migration SQL to get the database in the desired state (which is defined in Prisma schema).

The commands that suggest a reset are explicitly designed for use with development databases, not with production. The "deploy migrations to production database" command on the other hand only applies already existing migration files (that you have reviewed before) and does not suggest a reset, ever.

Re: Incident report for February 21st, 2024

#54

Post by pistoriusp »

namaria wrote: post #42
pistoriusp wrote: post #29

Unfortunately these sort of mistakes are seen as a "right of passage" for many developers. I ran "`DELETE FROM users;` without a WHERE clause against production in my first year on the job. I felt absolutely terrible. I thought I was connected to a development machine. Fortunately we had backups available. Often this isn't a problem with the individual developer itself, but points to a problem with the organization.…

I think you mean 'rite of passage'

Indeed. Thanks for the heads up.

Re: Incident report for February 21st, 2024

#55

Post by pistoriusp »

doctor_eval wrote: post #43
pistoriusp wrote: post #30

Earlier quoted context omitted.

Or just use https://www.snaplet.dev

Unclear how that would have helped.

My assumption is that they connected to production to run their local environment against production to debug something. They needed the data. Snaplet can capture a small subset of data related to a particular user, and remove the private information, which they can restore on their local machine.

Does that make sense?

Re: Incident report for February 21st, 2024

#56

Post by pistoriusp »

tianzhou wrote: post #33
pistoriusp wrote: post #29

Unfortunately these sort of mistakes are seen as a "right of passage" for many developers. I ran "`DELETE FROM users;` without a WHERE clause against production in my first year on the job. I felt absolutely terrible. I thought I was connected to a development machine. Fortunately we had backups available. Often this isn't a problem with the individual developer itself, but points to a problem with the organization.…

We are also building Bytebase, which enforces the change review process for such operations

It looks like we’re getting downvoted, lol. Not sure why that’s a thing. Bytebase is awesome. We should write about each other in our docs. You do the migration and we’ll do the data.

Re: Incident report for February 21st, 2024

#57

Post by kgeist »

seanwilson wrote: post #50
inopinatus wrote: post #12

Earlier quoted context omitted.

Someone writes the migration, commits it, it passes the build and unit test stages of the pipeline, then the application as currently running passes all function and integration tests with (and this is important) both the prior and the revised schema. Your commit is tagged as release ready! Not long after, the automation tooling confidently executes the now-tested migration under machine control during the next deplo…

> Someone writes the migration, commits it, it passes the build and unit test stages of the pipeline, then the application as currently running passes all function and integration tests with (and this is important) both the prior and the revised schema. Your commit is tagged as release ready! Not long after, the automation tooling confidently executes the now-tested migration under machine control during the next dep…

At our company, we have "an immutable DB", too, but when there's a critical emergency (say, full downtime), we can apply fixes manually. In that case, we run the tests after applying the fix.

Re: Incident report for February 21st, 2024

#58

Post by kgeist »

tianzhou wrote: post #33
pistoriusp wrote: post #29

Unfortunately these sort of mistakes are seen as a "right of passage" for many developers. I ran "`DELETE FROM users;` without a WHERE clause against production in my first year on the job. I felt absolutely terrible. I thought I was connected to a development machine. Fortunately we had backups available. Often this isn't a problem with the individual developer itself, but points to a problem with the organization.…

We are also building Bytebase, which enforces the change review process for such operations

I think DB engines should simply have some sort of a default option in the interactive mode where if you write "delete from table", it asks "Are you sure? You're going to wipe out the entire table! Y/N". Would've probably solved 99% problems.

Re: Incident report for February 21st, 2024

#59

Post by doctor_eval »

pistoriusp wrote: post #55
doctor_eval wrote: post #43

Earlier quoted context omitted.

Unclear how that would have helped.

My assumption is that they connected to production to run their local environment against production to debug something. They needed the data. Snaplet can capture a small subset of data related to a particular user, and remove the private information, which they can restore on their local machine. Does that make sense?

Sure; but I assumed they logged into production to run some queries to debug an issue unrelated to dev.

I mean, I know there are some bad practices out there - but connecting local dev environment to prod database server would be insane for any reason!

Re: Incident report for February 21st, 2024

#60

Post by PeterZaitsev »

You should always be ready for your database to get trashed - application bugs, operator error, hacker intervention...

What strikes me in the incident report they focus on failed migration, where the real issue is not planning or not testing for recovery if migration goes very wrong.

Even if backup recovery would take just 6 hours would it be acceptable ?

Post reply on HN