Live data from Hacker News

PostgreSQL example of self-contained stored procedures

sivers.org

11–20 of 81 posts

Re: PostgreSQL example of self-contained stored procedures

#11

How would somebody implement this for a team with versioning, source code view, and deployment?

We keep our schema in an XML file which is kept our source control repository alongside the application code.

We have a tool to generate the XML and to take the XML and compare with a database and altering it to match the XML schema. Though we never delete data due to backwards compatibility, so no dropping of objects, reduction of column widths etc.

When deploying a new version the database is processed first, if successful the new executable is made available. If not it calls home so we can take a look.

To make changes in development, we commit a new XML to dev branch, we have a commit hook to upgrade the dev database.

Re: PostgreSQL example of self-contained stored procedures

#12

How would somebody implement this for a team with versioning, source code view, and deployment?

If you are using SQL Server, you can create a SQL Server project in Visual Studio and keep all of the source code for your database objects (tables, views, functions, procedures, etc) in git.

The tools in Visual Studio will also help you create migration scripts and do schema/data diffs.

Re: PostgreSQL example of self-contained stored procedures

#13
post #3

I'm curious if anybody has practical experience developing a reasonably trafficked website in a similar manner. I wrote apps which consisted mostly of oracle stored procedures when I first got out of college, and it was a pretty awful experience, but it was also at a place where the development knowledge was minimal. I've since been getting closer and closer to writing postgres in this way. My current app (which I ar…

> I'm curious if anybody has practical experience developing a reasonably trafficked website in a similar manner.

I personally didn't develop but I've talked to people who did. The Estonian national web portal `eesti.ee` intially ran on stored procedures (and technically survived the DDoS of 2007 but was taken down "just in case" :S), later some Java garbage layer was added which significantly worsened the performance and made further development significantly harder.

Re: PostgreSQL example of self-contained stored procedures

#15

This is how I do relational databases in 2019. My database is a self contained module more akin to an rpc api than a database. My code has virtually no database logic in it at all. I don't know but it feels like functional databases could be a thing. We should call Martin Fowler and ask.

A few years ago I wrote an accounting software using that approach. The C++ code was calling predefined set of stored procedures and most logic (especially constraints) was in db. Fixing bugs not related to GUI was easy and, in most cases, not visible to end users. We also had a request to let an external software to insert some data into certain tables. I just created the procedure for this particular case and guys from another company just integrated their software without any problems.

Re: PostgreSQL example of self-contained stored procedures

#16

This is how I do relational databases in 2019. My database is a self contained module more akin to an rpc api than a database. My code has virtually no database logic in it at all. I don't know but it feels like functional databases could be a thing. We should call Martin Fowler and ask.

SQL is already a pretty functional language. It's a nice low statefulness, transactional layer around a very stateful core of data. Fully exploiting this gives you many benefits. This is the reason noSQL databases tend to be subpar.

Also, huge are the performance benefits of not always pulling data across multiple layers of cache, across the network, back into multiple layers of cash of another computer to be processed by some script running far away from the original data. Processing and aggregating as much as you can as you are retrieving the data is often multiple orders of magnitude faster and more efficient than sending everything elsewhere to be processed.

Re: PostgreSQL example of self-contained stored procedures

#17
post #3

I'm curious if anybody has practical experience developing a reasonably trafficked website in a similar manner. I wrote apps which consisted mostly of oracle stored procedures when I first got out of college, and it was a pretty awful experience, but it was also at a place where the development knowledge was minimal. I've since been getting closer and closer to writing postgres in this way. My current app (which I ar…

Every once in a while I encounter somebody who heavily promotes that way of working. There are a couple of arguments I've heard in favor of it. Often it's a mix of performance, ultimate data integrity, and "simplicity". I just don't buy it.

If performance is so critical, whatever you gain by moving everything to the database is lost by the fact that databases are harder to scale. You may be able to get by with a single instance a little longer, but when eventually you don't anymore, you've now worked yourself in a corner. It's an 80/20 game anyway, so just use an occasional stored procedure when it's really worth it.

For data integrity, I've always found databases not expressive enough. Sanity checks, okay, but you can't capture a non-trivial domain in some custom data types, check constraints and foreign keys. Even if you introduce stored procedures that will be responsible for keeping everything proper, you need to go crazy with permissions to block circumventing those. Might as well build your own API then. (I do find it difficult where to draw the line when it comes to what to enforce in the database still.)

"But you don't need another API! Just use the database as one!" Then I ask how they do testing, and the answer basically is: "We don't make mistakes or we find out about them (in production) soon enough." That pretty much ends the discussion for me. Surely there must be ways to devise a basic testing framework for stored procedures, but why bother? I don't want to spin up a database just to test some logic. Never mind testing, what about refactoring?

From a theoretical point of view, I can see the potential, but practically...? The tooling isn't there, and perhaps that's for a reason. Maybe databases are just not supposed to be used that way. Despite the extra functionality they offer, I treat them mostly as a data store.

What am I missing? I would love to be proven wrong.

Re: PostgreSQL example of self-contained stored procedures

#18

This is how I do relational databases in 2019. My database is a self contained module more akin to an rpc api than a database. My code has virtually no database logic in it at all. I don't know but it feels like functional databases could be a thing. We should call Martin Fowler and ask.

How is that not how databases are typically used?

Re: PostgreSQL example of self-contained stored procedures

#19
post #5
post #3

I'm curious if anybody has practical experience developing a reasonably trafficked website in a similar manner. I wrote apps which consisted mostly of oracle stored procedures when I first got out of college, and it was a pretty awful experience, but it was also at a place where the development knowledge was minimal. I've since been getting closer and closer to writing postgres in this way. My current app (which I ar…

You may find Graphile interesting https://www.graphile.org/

I prefer https://hasura.io

Instead of JS, it's written in Haskell (end user does not write haskell). It's incredibly performant and has hooks for database events.

Another similar tool is http://postgrest.org/en/v6.0/

Re: PostgreSQL example of self-contained stored procedures

#20
I'm the author. Thanks to Ludwig for posting this here.

I've been doing all of my database work like this for 5+ years now, and love it. Many more examples here:

https://code.sivers.org/db-api/

It works great when you want to write simple shell scripts, too. They look as simple as this:

https://github.com/sivers/store/blob/master/getdb-example.rb...

People have been asking how I do unit testing. The main thing is to have a fixtures.sql file, which is a dump of data to test against: https://github.com/sivers/store/blob/master/store/fixtures.s...

Then in the setup before each unit test, just drop the database schema, and reload it:

https://github.com/sivers/store/blob/master/test_tools.rb?ts...

Examples of my API unit tests here:

https://github.com/sivers/store/blob/master/store/test-api.r...

And also testing things like database triggers, and private functions:

https://github.com/sivers/store/blob/master/store/test-db.rb...

Feel free to ask me anything here, or email me: https://sivers.org/contact

— Derek

Post reply on HN