Live data from Hacker News

Ask HN: What's your preferred way of ensuring complex database integrity?

news.ycombinator.com

51–58 of 58 posts

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#51

Earlier quoted context omitted.

Don’t do that, was in response to: But as soon as you add another service with write capability, you need to re-implement the validation logic. What if you forget? In context of application validation logic, the logic is only living in one place. Triggers are basically “spooky action at a distance”. It’s also much easier to promote, version, upgrade, and rollback application changes.

There are consistency guarantees that it's possible that you can't ensure with just application logic. Suppose you are running a booking system for ridesharing. When you want to lock-in a passenger's seat, it's possible that consistency can be violated when you do it at the application level: both A and B request a seat after a query returns that there is a seat available, then simultaneously lock-in their reservatio…

Suppose you are running a booking system for ridesharing. When you want to lock-in a passenger's seat, it's possible that consistency can be violated when you do it at the application level: both A and B request a seat after a query returns that there is a seat available, then simultaneously lock-in their reservation after seeing stale data.

This is a solved problem.

  function reserveSeat (customerid, seatid)
  {
   lock
  {
      updatedRows  = update seats set customerid = {customerid} where id= {seatid} and customerid is null 

      return updatedRows == 1
   }
  }

Yes pretend scarfaceScript takes care of sql injection vulnerabilities and “lock” makes sure that only one thread can enter the block at any given time.

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#52

Isn't that the primary purpose of transactions? I mean, every time you write to the database, you open a transaction, do you stuff and close it. It either works or it doesn't. If it doesn't, then it rolls back. Isn't that enough? I guess I am saying that for complex integrity checks, it should be the responsibility of the application's data layer. On the other hand what is meant by "complex"? Is it "complex" because…

Here's an example of a "complex" constraint. You have a restaurant reservation system, and reservations are stored as time intervals. Each restaurant has a capacity, and you need to ensure that a booking does not make it so that the restaurant exceeds capacity at any point of time. One way to resolve it with triggers is to have a stored procedure fetch all the records that intersect the new booking's interval, and pr…

Imho that's the only way. Anything else can end up in invalid state.

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#53

Earlier quoted context omitted.

Foregoing triggers in favor of validating data in the application layer can be dangerous. The problem is that you need to re-implement data validation on every path to the database. If you've got a single application server, and it's the only client that ever writes to the database, this might work. But as soon as you add another service with write capability, you need to re-implement the validation logic. What if yo…

But as soon as you add another service with write capability, you need to re-implement the validation logic. Then Don’t Do That (tm) It’s hardly ever a good idea to have multiple apps writing to the same set of tables directly.

It's hardly a good idea if you have a database that allows invalid state.

Otherwise it's a fantastic idea.

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#54
post #53

Earlier quoted context omitted.

But as soon as you add another service with write capability, you need to re-implement the validation logic. Then Don’t Do That (tm) It’s hardly ever a good idea to have multiple apps writing to the same set of tables directly.

It's hardly a good idea if you have a database that allows invalid state. Otherwise it's a fantastic idea.

An “invalid state” can mean a lot of things - including business rules that can’t be expressed by simple relational, constraints rules.

But what happens when you need to change something about the database. Isn’t it a lot easier to have all of the code in one place?

So the usual retort is to put all of the business rules in stores procedures.

Then you have an unholy mess of a database with triggers and stored procedures that are harder to modify, harder to unit test, harder to version and just an unmaintainable mess.

I’ve never heard a single developer say that they love maintaining a system with 100s of large stores procedures, triggers, etc.

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#55

Earlier quoted context omitted.

There are consistency guarantees that it's possible that you can't ensure with just application logic. Suppose you are running a booking system for ridesharing. When you want to lock-in a passenger's seat, it's possible that consistency can be violated when you do it at the application level: both A and B request a seat after a query returns that there is a seat available, then simultaneously lock-in their reservatio…

Suppose you are running a booking system for ridesharing. When you want to lock-in a passenger's seat, it's possible that consistency can be violated when you do it at the application level: both A and B request a seat after a query returns that there is a seat available, then simultaneously lock-in their reservation after seeing stale data. This is a solved problem. function reserveSeat (customerid, seatid) { lock {…

This is only possible when you have a row for each available seat, though. Consider the case where you don't want to represent each seat in your application.

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#56

Generally my rule of thumb is to keep the integrity checking as close to the data as possible. Usually that means integrating it into the schema/database (optionally with triggers), as long as that has proper tooling (versioning specifically). With more complex checks it might be necessary to build a periodically runnable check mechanism. At startup might be a good way to do it, or some sort of cronjob. I can imagine…

I’ve never heard someone say as the database grows in complexity - “I’m so glad we decided to use triggers everywhere”.

I haven't actually used triggers myself, and I do feel like they're too far away from the data to be visible/easy to understand.

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#57
post #35

I’ve yet to encounter DB level integrity problems that aren’t solved by a well designed, normalized schema combined with proper use of transactions, PKs, FKs, unique and not null constraints, etc. These problems probably exist, but they’re very rare. Much more rare than people THINKING they have a problem like this, hacking around it with poorly implemented application code, and ending up with data that has integrity…

We ran a large e-commerce company main site solely on a single (with HA) SQL Server database. (We’ve since added other DBs, and always had other reporting and analysis systems, but we were doing over $1BB/yr on a single transactional server.) There’s no “room” there for foreign key constraints nor pervasive use of transactions. We used limited transactions in payments area only (and eventually even retired the last o…

Curious to know what you mean by “no room.” As in the performance impact of FKs and extensive use of transactions was too high? In my experience (not insane scale, but around 1 million daily active users hitting DBs with TBs of data), the performance overhead of FKs and transactions is fairly minimal. Ad-hoc attempts to bring the same functionality almost always end up in a loss of referential integrity, corrupt/wrong data, lots of bug squashing, etc. Why waste all that dev time and introduce all those bugs when you can use battle tested approaches like FKs and transactions that can guarantee correctness, at only a small performance hit?

Re: Ask HN: What's your preferred way of ensuring complex database integrity?

#58
It mostly depends on what type of table it is, I am a Db2 DBA z/OS. From experience, PK is must for every table either composite or single column. FK not recommended for 24/7 high volume or transaction table, recommended to handle it via code for ease of maintenance.

Column should be NOT NULL WITH DEFAULT, nulls introduces unnecessary additional where clause checks.

Post reply on HN