Live data from Hacker News

Database Migrations

vadimkravcenko.com

61–69 of 69 posts

Re: Database Migrations

#61
post #41

Earlier quoted context omitted.

Your comment is written like you have a undisclosed stake in skeema.io

Founder of skeema.io here. GP is a fan and does not have a stake in the company. Skeema is used by several hundred companies, including GitHub, Twilio, and Etsy. We have a lot of fans in the MySQL community, and just because someone enjoys the product does not mean they’re a shill.

Thanks for the clarification, and I wasn’t suggesting he was a shill - it just sounded like it and typically such claims and prefixed with a disclaimer on HN

Re: Database Migrations

#62

Earlier quoted context omitted.

how does this work? you have the "real" database and this "api" database makes queries to the real database? how does that work?

In postgres, every object (table, index, function, view, etc.) lives in a "schema", which is better thought of as a namespace. I put low-level objects like tables into one or more schemas, then create an "API" schema with views, functions, procedures that operate on tables in other schemas. Then I only grant access to that API schema to the application users.

thank you so much!

Re: Database Migrations

#63

Earlier quoted context omitted.

I think #2 is a bit dated approach by now. Certainly there's a place for this approach, when you have a DBA who is acting as gatekeeper to a large DB and the application is stable and is probably a monolith, but this adds friction and comes at a cost. It also tends to encourage a style of design where more business logic goes into the DB instead of stateless services, which eventually bottlenecks your ability to scal…

I last applied this approach a few years ago at an IIOT startup, where we had billions of rows of sensor data. We needed it all in one place, but with a fairly varied set of users, like an application team showing customers various slices of their data and a data science team applying all kinds of analytics to the data. When I showed up we had a single Postgres database with a half-dozen different clients. One manage…

Yeah, definitely a case-by-case basis situation, but I think with "modern" apps this is not a great approach. The situation you describe walking into is definitely horrible and adding some discipline and especially a solid DBA or lead can mitigate a lot of damage. I've seen plenty of outages caused by some dev not understanding the query their ORM was generating taking down production at the worst possible time, and this approach can help.

This once was the "correct" way to do things, period. At least that's how I learned it originally. If you go a back a couple of decades, "users" in a RDBMS meant literally end users, using a client application to directly query the DB from their desktop machine and whatever permissions they had was what they had. In this environment locking down everything except stored procedures and views was the only way to go. (Interestingly, we've sort of seen a resurgence of this design recently with some of these PaaS based web applications where all the logic is living client-side somewhere in a React app.)

After that indoctrination I remember feeling very resistant initially to a lot of this and in particular the practice of just granting full read/write access to a service account and letting developers write whatever they wanted, but times change. We were working under different constraints at the time. If you have services scoped to a specific responsibility and they completely own their own data and don't have the situation you describe of many clients touching the same DB, a lot of the problems that the proc+views only pattern were intended to solve just go away on their own.

Re: Database Migrations

#65
post #29
post #15

Earlier quoted context omitted.

I don't think that's what's meant by #2... About 12 years ago I worked at a place where all the crud operations were stored procs (calling tables/views/functions as needed). The application was only granted execute on the specific schemas. That meant no direct access for crud. A number of Schema changes and query optimizations could be handled by updating the stored procs without having to recompile the application.

joins and subqueries are part of SELECT statements, not crud. the purpose of using views is for the SELECT side of the application, not the CRUD side.

You're describing the "R" in CRUD, my friend: https://en.wikipedia.org/wiki/Create,_read,_update_and_delet...

In the example above, the select statements were inside stored procedures, and applications were only granted permissions to execute those with appropriate parameters.

Re: Database Migrations

#66
post #50

Earlier quoted context omitted.

Great post, thanks for the insight. I have a question regarding this point: > 2. never expose real tables to an application. Create an "API" schema which contains only views, functions, procedures, and only allow applications to use this schema. Would a view-based indirection help with rollbacks? For example, in scenarios where a column is added/dropped, would it work to just join with a new relationship column? Roll…

Yes, this absolutely works. Rather than viewing it as a rollback enabler, I think of it as a way to try things while keeping an escape hatch. Instead of deleting a column from a table, you can remove it from the view, and only drop it from the table when you're really sure that's safe. But dropping a column from the view is an API-breaking change, so rather than updating the existing view I'd make a new view, possibl…

This is spot on. I was having this exact discussion last week about how best to "strangle" an older application and deploy the new replacement for an upcoming project.

Re: Database Migrations

#67
The pain of database migrations is what originally pushed me towards event sourcing. The database is read-only (but immediately mirrors changes applied to the event stream), and I can have multiple databases connected to the same stream, with different schemas.

This makes schema changes easy to perform (just create a new database and let the system populate it from the events), easy to do without downtime (keep the old database available while the new database is building), and easy to roll back (keep the old database available for a while after the migration).

The trade-off is that changing/adding event types now needs to be done carefully (first deploy code able to process the new event types, then deploy the code that can produce the new events), whereas a SQL database supports new UPDATE or INSERT without a schema change.

Re: Database Migrations

#68

The pain of database migrations is what originally pushed me towards event sourcing. The database is read-only (but immediately mirrors changes applied to the event stream), and I can have multiple databases connected to the same stream, with different schemas. This makes schema changes easy to perform (just create a new database and let the system populate it from the events), easy to do without downtime (keep the o…

This assumes your stream has the full event history … or that history/state is irrelevant, correct?

If not, did you leave out a ‘copy history from source to target’ step?

Re: Database Migrations

#69
post #68

The pain of database migrations is what originally pushed me towards event sourcing. The database is read-only (but immediately mirrors changes applied to the event stream), and I can have multiple databases connected to the same stream, with different schemas. This makes schema changes easy to perform (just create a new database and let the system populate it from the events), easy to do without downtime (keep the o…

This assumes your stream has the full event history … or that history/state is irrelevant, correct? If not, did you leave out a ‘copy history from source to target’ step?

Indeed, the event stream contains the full history of all events that have ever been produced during the lifetime of the application.
Post reply on HN