Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban 1+N in Django

suor.github.io

Post reply on HN
Print view View original

31–40 of 153 posts

Re: Ban 1+N in Django

#31

Post by readertime »

btown wrote: post #18

My Chaotic Good take on this: one could implement a qs.auto_fetch_deferred() that emits model instances with weak back-references to a WeakSet of all instances emitted, and on a deferred get on ANY instance, it prefetches that attribute onto ALL of the instances... so that it doesn't just complain, but actually fixes your 1+N issue. But here lies absolute madness...

I agree; Django (and many ORMs) have taken the opinion that if you load N instances of a model, and then you load a related field on 1 of them, you likely only want the 1 related field. If however they assumed that loading a related field implies you likely want all related fields, perhaps there’d be fewer instances of foot-gunning.

Re: Ban 1+N in Django

#34

Post by s17n »

If you're going to do this, you may as well simply not use an ORM. Which is definitely the solution that I'd recommend; they are just not actually a good idea.

Re: Ban 1+N in Django

#35

Post by lmm »

binarymax wrote: post #16

This is why I always advocated against ORMs. It’s so easy to fall into traps like this without even knowing it, and while you can work around it in some ORMs it is not obvious. Writing SQL is not that hard, and mapping the results to a type isn’t that hard either. So with an ORM you might end up saving several hours of work up front for lots of pain later.

It's extremely easy to write SQL queries that take a lot longer than they look like - an unindexed full table scan looks exactly the same as an indexed join, for example, whereas in a good ORM they will look different. So as far as I can see writing SQL manually is just extra drudgery for no real gain.

Re: Ban 1+N in Django

#36

Post by zzzeek »

binarymax wrote: post #16

This is why I always advocated against ORMs. It’s so easy to fall into traps like this without even knowing it, and while you can work around it in some ORMs it is not obvious. Writing SQL is not that hard, and mapping the results to a type isn’t that hard either. So with an ORM you might end up saving several hours of work up front for lots of pain later.

> Writing SQL is not that hard, and mapping the results to a type isn’t that hard either.

But... Then you've written an ORM.

Re: Ban 1+N in Django

#37

Post by Daishiman »

s17n wrote: post #34

If you're going to do this, you may as well simply not use an ORM. Which is definitely the solution that I'd recommend; they are just not actually a good idea.

The ORM provides a myriad other features, like adapters for every production database under the sun, query composition that is literally impossible in plain SQL, a reasonable interface to the admin and the ecosystem of Django apps, and above all: a logical interface that maps _business objects_ to their SQL tables.

The ORM hate seems to come from people whose day to day interaction with data tables isn't mediated by seeing them as business entities where all the goodness of object-oriented programming actually shines.

Re: Ban 1+N in Django

#38

Post by Rapzid »

paulddraper wrote: post #33

In my experience, the far more pernicious way to get N+1 is serializers (Django REST)

Huh, ran into same issue on a rails project; queries kicked off in inside serializes were a weed. We toyed with throwing errors if a query was made during serialization.

On top of that, everything about serialization is terribly slow in Rails :|

Re: Ban 1+N in Django

#39

Post by nickjj »

mattbillenstein wrote: post #28
nickjj wrote: post #7

There is a case where having N+1 queries are beneficial. In Rails terms, it's when you perform Russian doll caching, but you can do this in any framework. The idea is you can cache a specific X thing which might make a query to an associated Y thing. A textbook N+1 query case (ie. a list of posts (X) that get the author's name (Y)). If you render the view without any cache with 10 things then you'd perform 20 queries…

Now you have like three problems instead of one - N+1 queries in the cold-cache case is slow, cache invalidation when something changes, and much more overall complexity...

> cache invalidation when something changes, and much more overall complexity

With Rails this style of caching is built into the framework, it's not too bad. It can be applied to other frameworks too because most of the "magic" is how the cache key is composed. Lots of frameworks have the bits and pieces to pull it off.

Here's a snippet from the Rails docs on caching[0]:
That will automatically produce cache keys that look like this:

    views/products/index:bea67108094918eeba42cd4a6e786901/products/1
The key is scoped to the view responsible for rendering the snippet of HTML and each product. If any attribute of that product changes that would result in different HTML being produced then the cache will be busted automatically by Rails because that hash in the middle of the cache key will be different. You can also "help" Rails when it comes to associations by adding a `touch: true` argument to your relationship once at the model level so Rails knows to bust the cache if the association's attributes change.

Like everything with caching, it's not 100% bullet proof and perfect but if you know the rules of how the framework works you can solve 95% of your caching needs without much headache. For the other 5% you have options but you can keep them on a need to know basis and look them up when you encounter issues (you may never need them in the end).

[0]: https://guides.rubyonrails.org/caching_with_rails.html#fragm...

Re: Ban 1+N in Django

#40

Post by Suor »

nickjj wrote: post #39
mattbillenstein wrote: post #28

Earlier quoted context omitted.

Now you have like three problems instead of one - N+1 queries in the cold-cache case is slow, cache invalidation when something changes, and much more overall complexity...

> cache invalidation when something changes, and much more overall complexity With Rails this style of caching is built into the framework, it's not too bad. It can be applied to other frameworks too because most of the "magic" is how the cache key is composed. Lots of frameworks have the bits and pieces to pull it off. Here's a snippet from the Rails docs on caching[0]: That will automatically produce cache keys tha…

cacheops can do the same, but it never came to my mind to actually go this way. I would rather write some special code to fetch with `id__in=[...]` and then cache individually.
Post reply on HN