Live data from Hacker News

Do you really need foreign keys?

shayon.dev

131–140 of 179 posts

Re: Do you really need foreign keys?

#131

Earlier quoted context omitted.

Whenever anyone talks about “the application,” I immediately ask “what about all the other applications?” I promise you that you will find customer service and accounting and biz dev have also built stuff that uses the database to get their jobs done (probably not with the same ORM or even the same language) unless you have taken draconian measures to prevent them.

My assumption (with modern applications!) is that nothing but the role directly owning the data will access the data. The development and DBA teams will likely have a role they can assume after performing a carefully-audited breakglass procedure to use in an emergency (rare) or to fulfill audit tasks. At least in my org this is a well-known problem with legacy applications sharing databases. Limit access to the datab…

When your data has outlived several generations of front-end technologies (mine started with Powerbuilder), you will find that ownership of the data does not control the evolution of how it will be used with what replaces past clients.

One particularly interesting Oracle problem is:

  ORA-00060: deadlock detected while waiting for resource 
Tom Kyte's book, Expert One-on-One Oracle, describes the primary culprit:

"Oracle considers deadlocks to be so rare, so unusual, that it creates a trace file on the server each and every time one does occur... The number one cause of deadlocks in the Oracle database, in my experience, is un-indexed foreign keys."

For another perspective, add to this a default setting in every SQLite database:

  $ sqlite3 verynew.db
  SQLite version 3.34.1 2021-01-20 14:10:07
  Enter ".help" for usage hints.
  sqlite> .dump
  PRAGMA foreign_keys=OFF;
  BEGIN TRANSACTION;
  COMMIT;
Foreign keys can cause interesting problems, and SQLite specifically prefers to avoid them.

Re: Do you really need foreign keys?

#132

Earlier quoted context omitted.

The main advantage is it's WAY faster if you are writing a lot. The big disadvantage of foreign keys is they do verify integrity. That means making lookups on every write/update which can be very costly especially as the model becomes more complex. If you have a read heavy application with low levels of writes then by all means put in foreign keys to you hearts content. But if you are at a point where you have millio…

But you pay the cost in checking it in the application, as GP said. If so, it simply is moving the cost from db to application layer. Is there a reason the checks can be implemented more efficiently in the application than the DB can?

Referential integrity problems usually happen due to missing deletes, improper deletes, or references that should be cleared.

The overhead of checking for the existence of referred to records in ordinary inserts and updates in application code is unnecessary in most cases, and that is where the problem is. Either you have to check to have any idea what is going on, because your key values are being supplied from an outside source or you should be able write your application so that it does not insert random references into your database.

If you actually need to delete a row that might be referred to, the best thing to do is not to do that, because you will need application level checks to make the reason why you cannot delete something visible in any case. 'Delete failed because the record is referred to somewhere' is usually an inadequate explanation. The application should probably check so that delete isn't even presented as an option in cases like that.

Re: Do you really need foreign keys?

#134

Foreign keys allow for pushing a really vital piece of business logic down to the database itself, referential integrity. This can be done in application logic, but that risks bugs allowing broken references into your data. Its more foolproof when these checks are enforced directly at the db, making sure data is valid before it's stored or updated.

There was an interesting topic on r/experienceddevs [1], where the dev team was arguing with some DBAs on adding another column. The DBA insisted on using FKs over enum / application level logic, etc. The comments there present some excellent arguments on top of everything you said.

[1] https://www.reddit.com/r/ExperiencedDevs/comments/18ldexi/wr...

Re: Do you really need foreign keys?

#135

Earlier quoted context omitted.

Whenever anyone talks about “the application,” I immediately ask “what about all the other applications?” I promise you that you will find customer service and accounting and biz dev have also built stuff that uses the database to get their jobs done (probably not with the same ORM or even the same language) unless you have taken draconian measures to prevent them.

Interesting that this whole thread makes no distinction between read and write access, as those are dramatically different use cases. Read access is by far the more necessary, and is usually solved relative easily by saving snapshots to a data warehouse. This is no panacea as data can still easily be misinterpreted or replicated and used out of context in violation of expected production data lifecycle by the team th…

Indeed - I immediately thought of hypothetical system (similar to many systems I’ve worked on) where SELECTS are 100x more common than INSERTS which are 100x more common than DELETES.

The 5x speed up on delete performance is a useless optimization.

Re: Do you really need foreign keys?

#136

A better post than I expected. The only thing I'd add is that foreign keys can actually improve read times because the optimiser knows it can safely skip certain joins e.g. if you have inner equi-joins between tables a, b and c a join b join c If there is an FK from a to b, and likewise from b to c, and you don't use anything in b, then the optimiser can rewrite this to a join c YMMV

Yes, I remember using foreign key relationships to optimize symmetric hash join back when I worked on an OLAP db. The idea was that for a 1:1 relationship you can immediately discard both sides of a joined tuple, while for a 1:many relationship you can immediately discard the "many" side of a joined tuple.

Re: Do you really need foreign keys?

#137

Earlier quoted context omitted.

Whenever anyone talks about “the application,” I immediately ask “what about all the other applications?” I promise you that you will find customer service and accounting and biz dev have also built stuff that uses the database to get their jobs done (probably not with the same ORM or even the same language) unless you have taken draconian measures to prevent them.

+1. When I started in the industry, it was common for more experienced developers to drill the “data outlives the application that generated it” principle into you. Somewhere in the transition to NoSQL and back we lost this.

It wasn't the transition to NoSQL. It was the transition to web APIs. Now the application that can access the database has a universal shim in place that deals with the messy business of taking requests and applying them to the data store.

Now there's typically three applications that need DB access:

1. The application it was built for.

2. The read-only reporting and visualization tools.

3. The web API.

Re: Do you really need foreign keys?

#138

Earlier quoted context omitted.

What would be the advantage of implementing it in the application? I only see disadvantages: - no data integrity check in the database - more complicated definition of foreign relation, or none at all, leaving possibility for deviant data - scatteted schema. Cant look at the db table and understand the entire model. Have to hunt in code an potentially across a lot of code. It just seems like a data integrity issue th…

The main advantage is it's WAY faster if you are writing a lot. The big disadvantage of foreign keys is they do verify integrity. That means making lookups on every write/update which can be very costly especially as the model becomes more complex. If you have a read heavy application with low levels of writes then by all means put in foreign keys to you hearts content. But if you are at a point where you have millio…

Batch processing is also WAY faster (like 1-2 orders of magnitude), but for whatever reason people don't talk about it much (cynically, I guess because it's too straightforward and old-hat). As far as I can tell, Rails doesn't have multithreading/async, so the OP is presumably not batching requests. That seems like a much better place to start than giving up referential integrity.

Re: Do you really need foreign keys?

#139
post #101

Earlier quoted context omitted.

If you operate at the scale where you consider the performance implications of foreign keys, you probably are not ok with anyone accessing the database or running just any query against it. It is not realistic that you can trust everyone who needs to access the data with access to the database as they might easily cause problems with poorly written queries. Additionally you may want invariants maintained, that the da…

> If you operate at the scale where you consider the performance implications of foreign keys, you probably are not ok with anyone accessing the database or running just any query against it. I used to work on an application where ALL database accesses were via stored procedures. Genuinly the best dev experience, to me, so far.

> I used to work on an application where ALL database accesses were via stored procedures. Genuinly the best dev experience, to me, so far.

I found such an environment to be simply terrible.

In general, I think stored procs certainly have their place, but if ALL access is through stored procs, you better have a schema that’s basically set in stone otherwise dev will turn into a nightmare.

Re: Do you really need foreign keys?

#140
post #97

Follow this advice with caution. Dropping foreign keys is effectively giving up part of the C in ACID. It should be done with very, very open eyes to the downsides. I'm not sure the author is selling the "when" side of this very well. Migrations are "hard" because the database is forcing you to handle correctness criteria that are easy to ignore. "Lock contention" is the database covering your sloppy ill-thought out…

That's the best advice: data will outlive the application. It will also serve other applications. So treat it as its own thing, not as an appendix of the application.

Imo better advice would be your data may outlive the schema it's stored in or your application may outlive the data.
Post reply on HN