Live data from Hacker News

PostgreSQL example of self-contained stored procedures

sivers.org

31–40 of 81 posts

Re: PostgreSQL example of self-contained stored procedures

#31
post #28
post #20

I'm the author. Thanks to Ludwig for posting this here. I've been doing all of my database work like this for 5+ years now, and love it. Many more examples here: https://code.sivers.org/db-api/ It works great when you want to write simple shell scripts, too. They look as simple as this: https://github.com/sivers/store/blob/master/getdb-example.rb... People have been asking how I do unit testing. The main thing is to…

I read this a few days ago. I was waiting for it to show on HN because I have a few questions. I want to do the same, but with big databases (the TXID wraparound is a problem every few months). For now, the queries are stored in the code. They rarely evolve. How do you manage versioning? Queries in code mean you can ensure everything match. How easily other people interact with that? Not many developers know SQL well…

[deleted]

Re: PostgreSQL example of self-contained stored procedures

#32
post #28
post #20

I'm the author. Thanks to Ludwig for posting this here. I've been doing all of my database work like this for 5+ years now, and love it. Many more examples here: https://code.sivers.org/db-api/ It works great when you want to write simple shell scripts, too. They look as simple as this: https://github.com/sivers/store/blob/master/getdb-example.rb... People have been asking how I do unit testing. The main thing is to…

I read this a few days ago. I was waiting for it to show on HN because I have a few questions. I want to do the same, but with big databases (the TXID wraparound is a problem every few months). For now, the queries are stored in the code. They rarely evolve. How do you manage versioning? Queries in code mean you can ensure everything match. How easily other people interact with that? Not many developers know SQL well…

You just talked about TXID wraparound, materialized views and then talking about not many developers know SQL well. You're all over the place with your questions.

This is very simple, you wrap all your logic as functions/stored procedures, have the DB do the work. This will work for 99% of businesses out there. How many people really wraparound ID? How many people really need to shard DB? Those 1% will know how to tackle the problem.

For most people by moving all these into the DB, your code is much simpler. Something like this should be encouraged more, it's far easier to implement something like this than deal with ORM.

Re: PostgreSQL example of self-contained stored procedures

#34
I’ve been using this pattern recently. Very early stages but I like it so far.

I wrote some boilerplate tools for easily loading schemata and extensions. This is a must have for being able to write isolated tests.

I keep all data definitions in schemata, which can be managed by standard migration scripts. I keep function definitions in extensions, which do not create any data stores but expose functions with well defined interfaces for querying the underlying data. I rely on standard Postgres extension loading to manage migrations.

I really like this pattern, but there is a degree of unapproachability to it. It takes a lot of effort to make sure tests work well - but it’s not so bad. Since your DB functions do everything, the app code is just a shell around those functions (few lines of code to call each function), so by testing those functions you are exercising their Postgres counterparts.

With tools like postgraphile and Postgrest available, this is becoming a more appealing option. What we need is more tooling for managing deployment, installation, migration, verification of and communication with the stored procedures from within application code. Combined with Postgres PL libraries like plv8, it should be possible to write JavaScript code within your application toolchain that you can test locally, but deploy to run in Postgres context.

Re: PostgreSQL example of self-contained stored procedures

#35
post #25

Why are stored procs still used? My experience with the black hole (or really I should say not in version control) pushed me away from them.

Why not put them in version control, and have your deployment process pull them out of the repo and apply them to the database?

yep

CREATE OR REPLACE FUNCTION ...

what whatever it was, it now is what came out of version control

Re: PostgreSQL example of self-contained stored procedures

#36
post #3

I'm curious if anybody has practical experience developing a reasonably trafficked website in a similar manner. I wrote apps which consisted mostly of oracle stored procedures when I first got out of college, and it was a pretty awful experience, but it was also at a place where the development knowledge was minimal. I've since been getting closer and closer to writing postgres in this way. My current app (which I ar…

Every once in a while I encounter somebody who heavily promotes that way of working. There are a couple of arguments I've heard in favor of it. Often it's a mix of performance, ultimate data integrity, and "simplicity". I just don't buy it. If performance is so critical, whatever you gain by moving everything to the database is lost by the fact that databases are harder to scale. You may be able to get by with a sing…

The stored procedure can operate on foreign schemas, so could be scaled that way. Then you would have a bunch of lightweight PG engines with no data in them making calls to the upstream.

In practice though, the logic your application uses is going to generate the same queries anyway. So I’m skeptical that using stored procedures increases the workload of the database.

If you need to scale your database, you can do it with any standard replication strategy, and just make sure to bring the functions to the replica too. If you package them in extensions, this is easy.

To be clear, you’re not answering web requests from the database. There is still an application in between; it’s just much thinner, or can be some abstraction like postgrest or postgraphile. You scale that too.

Re: PostgreSQL example of self-contained stored procedures

#37
This kind of approach can be good for getting data into the database, especially for non-trivial updates, but is not good for getting data out. The problem is if you want things like custom group-bys, sorts, joins, windowing, etc, you end up reinventing a lot of concepts, or copy+pasting procedures with slight variations. It basically takes SQL's biggest issue, query composability, and makes it even worse than it already is.

Re: PostgreSQL example of self-contained stored procedures

#38
post #29
post #28

Earlier quoted context omitted.

I read this a few days ago. I was waiting for it to show on HN because I have a few questions. I want to do the same, but with big databases (the TXID wraparound is a problem every few months). For now, the queries are stored in the code. They rarely evolve. How do you manage versioning? Queries in code mean you can ensure everything match. How easily other people interact with that? Not many developers know SQL well…

Versioning: sorry I'm not sure what you mean. Same as if you had queries in an ORM and views using the results, you'd need to branch and make changes, then merge the branch together at once. I just do the same. If the database structure changes, I run the ALTER TABLE commands, when switching to that branch. Other people : I guess like anyone choosing a language, you're excluding those who don't know it. I'm assuming…

Thanks a lot for your reply!

It seems the main gain for your is to move between languages and allow initial iterations/bugfixes without touching the application code.

The database has more maintenance issues (like rolling txids) than the application code. I am not sure I want to add more complexity and potential issues to the database.

FYI, a materialized view is a potentially long or complex query whose results are cached, so you can say 'select * from complex_query_result' to get them, and refresh the complex_query_result whenever you feel like it. You can also update the query that generates complex_query_result.

In practice, MV can give you speed (as you can refresh the MV when you need/want, while keeping the results) and also put the data-logic inside the database (as the MV is defined initially, and can later be updated) if you don't need super fresh results. If you do, use a regular view.

In either case, you can use the view approach when parameters are needed, iff you can reduce you query to where parameter=something on the view. Otherwise, you need to use languages like pl sql.

As the materialized views queries just return the results to be processed, and I have very little extra to do, your approach seems overkill for my use case.

Re: PostgreSQL example of self-contained stored procedures

#39
post #25

Why are stored procs still used? My experience with the black hole (or really I should say not in version control) pushed me away from them.

Why not put them in version control, and have your deployment process pull them out of the repo and apply them to the database?

It's a bit more complicated than just "upload all my stored procedures" in practice. I think we need a version control / deployment system that is designed for databases to make this work.

We want it to regard the database as the source of truth, and understand that while you can blow away a development database, you can't do that to prod.

We want it to be able to track which clients are using which stored procedures so it can prohibit dropping stored procedures that are in use. Ideally, when those clients upgrade to newer versions, it's then able to drop them automatically.

Stored procedures can bind to tables, views, types, etc. so it needs to be able to recommend migration plans.

Re: PostgreSQL example of self-contained stored procedures

#40
post #20

I'm the author. Thanks to Ludwig for posting this here. I've been doing all of my database work like this for 5+ years now, and love it. Many more examples here: https://code.sivers.org/db-api/ It works great when you want to write simple shell scripts, too. They look as simple as this: https://github.com/sivers/store/blob/master/getdb-example.rb... People have been asking how I do unit testing. The main thing is to…

Pretty much how we built client-server applications on Oracle RDBMS 25 years ago.

When your application grows, you'll have a large number of stored procedures. Oracle has packages which significantly eases the way you work with a large amount of stored procedures. Packages enable you to group related stored procedures into one object.

Packages have an interface (stored procedures and custom types available outside the package), and an implementation (package body). You can manage security at the package level, and you can hide the implementation of the stored procedures in the package body.

The postgres alternative, to use schema, isn't very good: a schema is more like a package in java, an oracle package like a class. Doing java with only packages, and no classes feels very limited.

Post reply on HN