Live data from Hacker News

We do not use foreign keys (2016)

github.com

321–330 of 337 posts

Re: We do not use foreign keys (2016)

#321
while most rdbms-based projects will continue using foreign key constraints (and all will be referentially cast in stone), and postgressives will enjoy 100000x query speedups by remembering to include your hornsnaggle_tree_index_lookup_speeder_upper index,

there is a certain fanatical mindset from the rdbms world that sounds a bit like:

never forget always include jquery with every web app you make. it's made by real profeshunalz who know more than you and lots of people already included it and you should simply just learn to accept that all front-end web problems have already all been solved by jqueremy and you should simply refactor it all in terms of jqueremy. anything else is korrupshun. In other words, always only ever consider the same tools and techniques. LOL

A mere search for other database types produces pages upon pages of "why sql is beating nosql" and all stack overflow answers always doom you to the 7th level of hell for anything but 3rd normalized form. hey let's all use lookup tables for many to many. it's a cult. it's nuts. anyone who has spent any amount of time with database-backed applications is surely aware of the tradeoffs they are making when choosing non-relational databases of any kind.

The obsession with forcing every all all use cases into the rdbms mindset makes me say "well then, let's just flatten everything even further into mapped key-value a la ohm." in which one encodes the key with path info and the value with the value, etc.

if the act of suffering for the sake of suffering and breaking things into iddy bidd y pieces is virtuous in and of itself, surely this is a more true pure and rigorous religion to bend every single thing to.

(before smarty pants says "um, querying" may I remind you of adding another entire secondary index of any kind via adding it with the id of the first key as it's value.)

I'm not advocating everyone do this. no freakouts pls.

ideally we would just persist our objects, which is coming with persistent memory, but i digress...

This Lance fellow seems to have taken plenty of "you have no idea what you are talking about moron!" abuse, despite having a phd in computation theory and writing erp software since the dinosaurs roamed the earth. (I'm also not advocating his solutions for any and all problems, but not taking his ideas seriously is a mistake IMO) https://codeburst.io/databases-the-future-buggy-whips-of-sof...

https://codeburst.io/doing-without-databases-in-the-21st-cen...

mongodb is an incredibly powerful tool for when you want the whole json, all the json, and packed as bson. When you want to store/dish out blobs and not join nor have no use for the parts separately, then you can do that nicely.

(besides this usual story about "for when you don't know your schema or have lots of irregular semi-structured data, stripe, your mailgun webhooks, whatever. It's a great DB, for some things, despite the people saying "since it didn't do this one thing i really obsess over and think all dbs should always do that therefore its' crap"

(well, copying the same master-slave with single-write master failover or vertically scale story as the rdbms people was probably a weak point...)

pg as a jsonb store with secondary indices on actual columns LOL...

I guess we just basically need to accept that the same tools can be used as primitives/components in different contexts in terms of the actual application.

Some people use Redis as a primary datastore.

zealotry is best avoided, I guess..

look, this silo-ing stuff is also silly, and a product of the corporate dysfunctional mentality: split everyone up and get DBA's warring with Devs...

Re: We do not use foreign keys (2016)

#322

while most rdbms-based projects will continue using foreign key constraints (and all will be referentially cast in stone), and postgressives will enjoy 100000x query speedups by remembering to include your hornsnaggle_tree_index_lookup_speeder_upper index, there is a certain fanatical mindset from the rdbms world that sounds a bit like: never forget always include jquery with every web app you make. it's made by real…

http://www.dbta.com/Columns/Database-Elaborations/The-Refere...

Re: We do not use foreign keys (2016)

#323

while most rdbms-based projects will continue using foreign key constraints (and all will be referentially cast in stone), and postgressives will enjoy 100000x query speedups by remembering to include your hornsnaggle_tree_index_lookup_speeder_upper index, there is a certain fanatical mindset from the rdbms world that sounds a bit like: never forget always include jquery with every web app you make. it's made by real…

oh, I also forgot to add that many applications mix-n-match data components to create a data layer with a standard API that then may then be implemented with various backends for various purposes, each being applications in their own right, etc.

this is seldom discussed in such debates but is actually quite common for many applications.

Re: We do not use foreign keys (2016)

#324
post #256
post #211

Earlier quoted context omitted.

Will result in corruption. Yes. Will result in more fault tolerant software, also yes. It's a trade off and one I make willingly at every scale. I stopped using foreign keys after university and have never wanted them since. Non nullable database fields are far more useful than worrying about fks.

> Will result in more fault tolerant software More fault tolerant because you have to waste time debugging the faults and monkeypatching them in code just to avoid FKs? > Non nullable database fields are far more useful than worrying about fks Can’t even count how many non-nullable fields I’ve seen packed with “” empty strings to get around that requirement

1. Over my career almost none of it has been debugging problems due to a lack of foreign keys.

Further to that, instead of having lazy cascading deletes moving the goal posts on you. Data missing its parent for example is an indication something is wrong. It's a useful diagnostic.

2. Most string data is safe to represent as empty. Handling null on the other hand is a different situation and often induces warnings or other side effects depending on the language. Forcing the null checking to your boundaries is much like forcing your state/mutation to the boundaries in FP. Hell eveb though I think it's insane, hexagonal arch works on this idea as well.

Re: We do not use foreign keys (2016)

#325

Earlier quoted context omitted.

> These companies don't use MySQL directly. They use it indirectly as the storage component of a larger architecture. Yes and no. Often it's both. I say this first-hand, having performed significant work on the database tier for two of the companies I listed, and consulted for several others. For example, while Facebook's largest db tier goes through a dao / writethru cache, there's plenty of other use-cases that are…

> For example, while Facebook's largest db tier goes through a dao / writethru cache, there's plenty of other use-cases that are direct MySQL usage. I didn't know about that. That's interesting! > And in any case, why does it matter if there's another layer involved? I was writing that in the context of the parent comment about "disabling constraints". I can see why disabling constraints makes sense in a sharded envi…

To explain more re: FB and having another layer on top of MySQL, there are a bunch of separate sharded MySQL tiers there. It's split by workload -- for example, the access pattern, schema, and sharding key differs completely between the main social graph, Messenger data, ad market, financial transaction data, etc. And then there's also the internal MySQL database-as-a-service, which allows any engineer to provision one or many databases for any other purpose. Overall, some of these things have services on top that use MySQL more as low-level storage, and others use MySQL in a more traditional fashion.

re: "disabling constraints", that's kind of orthogonal. The large MySQL users simply don't create foreign key constraints in the first place; there's nothing to disable :) Whereas MySQL's ability to disable constraints for a single session is a feature intended to make things like logical dump/restore easier and faster, schema management easier, etc. Without that feature, these tools would need to construct a dependency graph and create tables in a specific order (and/or defer FK creation until after the tables), which is needlessly complex if the tables are new/empty, and very slow if restoring a logical dump which is already known to be referentially consistent.

As for MySQL vs Postgres, IMO both databases are close enough in major functionality that for many use-cases it's best to just go with what you already know, can hire for, and can operate. There are special cases where one is better than the other, for example personally I'd go with MySQL for social networking / UGC / very high volume OLTP, and go with Postgres for use-cases where solid geospatial, OLAP, or fulltext are core requirements and/or there's a desire to minimize the number of different data stores.

Ideally with MySQL you're just using it for OLTP, and deferring to separate systems for OLAP, fulltext search, etc. In a way that's "more UNIXy" but it's also potentially an operational headache.

In terms of specific feature comparison, you already have a great list there. A couple other things on the MySQL side I'd mention are InnoDB's buffer pool (smarter caching than relying on the OS cache as pg does) as well as the existence of MyRocks storage engine (LSM-based system offering better compression than pretty much anything else of comparable performance level for OLTP).

That all said -- Postgres is an awesome database, and I'd say that Postgres is more closely aligned with the textbook definition of a proper relational database. But then again I'd also say something similar about FreeBSD (vs Linux) for server operating systems, yet for practical purposes I always go with Linux anyway :)

Re: We do not use foreign keys (2016)

#326
post #305

Earlier quoted context omitted.

If you ain't testing against the full system state, then I'd be hard-pressed to call that "integration testing". If you're generating the test data from scratch, then it shouldn't be hard to generate the dependent data while you're at it. If you're testing against (anonymized) production data, then it shouldn't be hard to pull the dependent data while you're at it. In either case, you should be validating the integri…

It isn’t hard, but often very cumbersome. Oh, so you want to test invoicing, for this Customer, which must have a Delivery Address and an Invoicing Adress which both needs valid Postal Codes. And the Customer must have a Contact person. Then we need the Product, that must consist of at least one Article, and each Article must be connected to the Company that we bought it from, with Addresses and Contact Persons and h…

That all seems pretty reasonable to me, and darn well should be included when "test[ing] invoicing":

- You surely want to make sure invoice creation depends on a valid billing address at the very least, right? If that breaks, you're gonna have a lot of rather irate AR clerks.

- You surely want to make sure the Contact is aware of the new invoice on the order, right? In fact, that might very well be part of the Contact's performance metrics, so if that breaks, you're gonna have a lot of rather irate sales/support reps (whatever "Contact" means in this context).

- You surely want to make sure your invoices are against valid items, right? And you surely want to make sure that expected revenue correctly ties back to an inventory movement, which in turn ties back to an inventory receipt, which in turn ties back to a paid invoice to a vendor, right? If that chain breaks, you're gonna have a lot of rather irate accountants and financial auditors.

- You surely want to make sure you're getting the right sales tax calculations, right? If that breaks, you're gonna have a lot of rather irate accountants, financial auditors, and tax collectors.

- You surely want to make sure the Sales Person and Sales Office both get credit for the invoice, right? If that breaks, you're gonna have a lot of rather irate sales reps and managers thereof.

So... no, I ain't exactly getting your point, lol. If you're changing invoicing, then all of the dependencies and dependents of invoicing ought to be tested.

But more to my point:

"very grateful for tools like tsqlt that make unit testing possible"

Unit testing != integration testing. If you're unit testing, then sure, turn off foreign keys and practice your quick draw while you cowboy it up. If you're integration testing, then that inherently means testing the whole system as a whole; what you call "cumbersome" I call "the bare minimum of comprehensiveness".

"I’ve easily used more than a full day just to get enough data in a naked system to make just the simplest test."

That's usually pretty easy to script, even with foreign key constraints. It might take you a day, but future days should be able to call upon that same script, saving you quite a bit of time :)

Re: We do not use foreign keys (2016)

#327
post #295

Earlier quoted context omitted.

If you're finalizing the invoice, you're hopefully doing something like this, right? UPDATE invoices SET final = TRUE, invoice_number = @InvoiceNumber WHERE id = @InvoiceId; (Where @InvoiceNumber is some variable the application's substituting into the query) If so, then the problem you present should never happen (unless the DB doesn't do atomic updates by default, but wrapping the update in a transaction should pro…

>you're hopefully doing something like this, right? Well, in an ideal case, yes, you would be doing it like that. The reality might be different, especially when using an ORM. >Of course, I'd also be wondering why an invoice would ever exist without an invoice number Draft invoices do not have an invoice number, since they don't really exist anywhere. You can delete a draft invoice and nothing has happened. But if yo…

> Well, in an ideal case, yes, you would be doing it like that.

I mean, it'd be either that specific case or a bug. And if it's a bug, then the check constraint as described previously would catch it and prevent it entirely.

Another option, though, would be to not even bother with a separate field for "final"; if the only two states are "draft" and "final", and finality is conditional on there being an invoice number, then the application logic can be greatly simplified:

    SELECT CASE
        WHEN invoice_number IS NULL THEN 'Draft'
        ELSE 'Final'
    END AS state
    FROM invoices WHERE -- yadda yadda yadda
Most databases can cache this as a computed value column on the table itself or in a view or what have you.

> Draft invoices do not have an invoice number, since they don't really exist anywhere.

Sure they do: they exist in your database. Unless your company has a strict rule about invoice numbers always being sequential with no gaps (and that'd be pretty darn strict, in my experience), you might as well pre-assign it.

Re: We do not use foreign keys (2016)

#328
post #165

Earlier quoted context omitted.

Thank you, this was helpful for me. I still don't, frankly, recall (did I ever know) how Rails implements the latter without the former.

No problem! The summary is basically this: if you say that a User has_many foos, then the foos table will have a user_id in it. When you ask for the foos a user has, the SQL will be emitted that has the "user_id=1" or whatever clause on it. If you add a foreign key constraint, the database itself will verify that the foos have valid user_ids. If you don't, then it won't. You're basically giving up the ability for the…

Yes, the word "constraint" is confusingly absent (or, rather, used colloquially) in the OP.

> FKs impose a lot of constraints on what's possible and what's not possible.

Thanks again for the further clarification.

Re: We do not use foreign keys (2016)

#329
post #305

Earlier quoted context omitted.

It isn’t hard, but often very cumbersome. Oh, so you want to test invoicing, for this Customer, which must have a Delivery Address and an Invoicing Adress which both needs valid Postal Codes. And the Customer must have a Contact person. Then we need the Product, that must consist of at least one Article, and each Article must be connected to the Company that we bought it from, with Addresses and Contact Persons and h…

That all seems pretty reasonable to me, and darn well should be included when "test[ing] invoicing": - You surely want to make sure invoice creation depends on a valid billing address at the very least, right? If that breaks, you're gonna have a lot of rather irate AR clerks. - You surely want to make sure the Contact is aware of the new invoice on the order, right? In fact, that might very well be part of the Contac…

I don't think we disagree about much. My only objection was to

> then it shouldn't be hard to generate the dependent data while you're at it

which makes in sound like somewhat light work. My point was only that it isn't. And of course we script a lot of this test-data-creation, but then you need different kind of data for different tests, and need to add some flexibility. And after a while, just generating test data becomes somewhat complex in itself. And all of the test-data scripts needs to be maintained and changed whenever the model changes. None of this is done in a breeze.

Re: We do not use foreign keys (2016)

#330

Earlier quoted context omitted.

> Maybe shitty toy database systems like MySQL While appropiate to define in few words some of MySQL's colossal mistakes, this isn't the kind of language that will sway heads that have been comfortably using MySQL because those defects are just "what DB's do".

Any database that would let you disable constraints on a session basis is a toy database. Such an operation doesn’t even make sense because at some point the relational integrity has to be enforced for the entire table. You can’t just have parts of a table be relationally correct. That is like saying 1 + 1 = 3. It is a completely illogical statement. However I would not at all be surprised to learn MySQL supports suc…

PostgreSQL is not a toy database either, and yet you can do:

    ALTER TABLE  DISABLE TRIGGER ALL
Or:

    SET session_replication_role = 'replica'
Post reply on HN