Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban 1+N in Django

suor.github.io

Post reply on HN
Print view View original

21–30 of 153 posts

Re: Ban 1+N in Django

#21

Post by stephen »

tveita wrote: post #13
nickjj wrote: post #7

There is a case where having N+1 queries are beneficial. In Rails terms, it's when you perform Russian doll caching, but you can do this in any framework. The idea is you can cache a specific X thing which might make a query to an associated Y thing. A textbook N+1 query case (ie. a list of posts (X) that get the author's name (Y)). If you render the view without any cache with 10 things then you'd perform 20 queries…

You'd ideally want to do something like dataloader, where you look up your N Xs in a single cache query, and then do a single database lookup for the (N-C) Xs that weren't in cache. You can then either eagerly load the Ys with the Xs like you said, or do a secondary cache lookup for every Y, and potentially another single database query for the Ys not in cache. Unfortunately this pattern gets really hairy if you're n…

+1. The JS event loop auto-monad-izing Promises into Haxl [1]-esqe trees of implicitly-batched loads has been a big win for us building on JavaScript/TypeScript.

If I had to move to another language, I'd really want to find a "powered by the event loop / dataloader" framework, i.e. Vert.x for Java.

Also, per dataloader, a shameless plug for our ORM that has dataloader de-N+1-ing built natively into all object graph traversals:

https://joist-orm.io/docs/goals/avoiding-n-plus-1s

[1]: https://github.com/facebook/Haxl

Re: Ban 1+N in Django

#22

Post by code_biologist »

On an unrelated note, Python folks should check out OP's library funcy [1]: "A collection of fancy functional tools focused on practicality. Inspired by clojure, underscore and my own abstractions."

Thanks for the library Suor! I've used it happily for many years!

[1] https://github.com/Suor/funcy

Re: Ban 1+N in Django

#23

Post by capableweb »

mattbillenstein wrote: post #19
binarymax wrote: post #16

This is why I always advocated against ORMs. It’s so easy to fall into traps like this without even knowing it, and while you can work around it in some ORMs it is not obvious. Writing SQL is not that hard, and mapping the results to a type isn’t that hard either. So with an ORM you might end up saving several hours of work up front for lots of pain later.

You're on the right side of the bell-curve meme my friend, but there are a lot more people in the thick-framework camp who spend their days getting lost in the complexity of ORMs and related tech...

It's more like the world is not black and white, engineering problems don't have "The perfect solution, the rest is trash", but rather "this problem has multiple solutions, depending on context, some have these tradeoffs and the others have these".

In this particular case, it might not be worth to trade speed of having to think about SQL for performance (today or tomorrow). Maybe you're building something that will just be used by 2-3 people, so 1+N isn't really a issue.

Or whatever, the conclusion as always is: it depends.

Re: Ban 1+N in Django

#24

Post by selcuka »

spapas82 wrote: post #14

This is very useful! I'm gonna start integrating it with my projects. However having a way to allow/not allow n+1 queries (like a context manager) would be much better. The thing is that there are times where n+1 isn't a big problem and fixing it would be a form of premature optimisation. I'd prefer to be in control and decide if I care about the n+1 query situation or not for some specific view.

It's not ideal, but you can simply log `DeferredAttribute.__get__` calls instead of raising an exception.

Re: Ban 1+N in Django

#26

Post by tantalor »

spapas82 wrote: post #14

This is very useful! I'm gonna start integrating it with my projects. However having a way to allow/not allow n+1 queries (like a context manager) would be much better. The thing is that there are times where n+1 isn't a big problem and fixing it would be a form of premature optimisation. I'd prefer to be in control and decide if I care about the n+1 query situation or not for some specific view.

Just keep a list of exemptions in a file

Re: Ban 1+N in Django

#27

Post by mattbillenstein »

capableweb wrote: post #23
mattbillenstein wrote: post #19

Earlier quoted context omitted.

You're on the right side of the bell-curve meme my friend, but there are a lot more people in the thick-framework camp who spend their days getting lost in the complexity of ORMs and related tech...

It's more like the world is not black and white, engineering problems don't have "The perfect solution, the rest is trash", but rather "this problem has multiple solutions, depending on context, some have these tradeoffs and the others have these". In this particular case, it might not be worth to trade speed of having to think about SQL for performance (today or tomorrow). Maybe you're building something that will j…

I agree it depends, but my hot take is almost universally the time people like to say ORMs save them they end up paying back in spades debugging them. Learn SQL!

Re: Ban 1+N in Django

#28

Post by mattbillenstein »

nickjj wrote: post #7

There is a case where having N+1 queries are beneficial. In Rails terms, it's when you perform Russian doll caching, but you can do this in any framework. The idea is you can cache a specific X thing which might make a query to an associated Y thing. A textbook N+1 query case (ie. a list of posts (X) that get the author's name (Y)). If you render the view without any cache with 10 things then you'd perform 20 queries…

Now you have like three problems instead of one - N+1 queries in the cold-cache case is slow, cache invalidation when something changes, and much more overall complexity...

Re: Ban 1+N in Django

#29

Post by cellularmitosis »

mattbillenstein wrote: post #19
binarymax wrote: post #16

This is why I always advocated against ORMs. It’s so easy to fall into traps like this without even knowing it, and while you can work around it in some ORMs it is not obvious. Writing SQL is not that hard, and mapping the results to a type isn’t that hard either. So with an ORM you might end up saving several hours of work up front for lots of pain later.

You're on the right side of the bell-curve meme my friend, but there are a lot more people in the thick-framework camp who spend their days getting lost in the complexity of ORMs and related tech...

I really think it is more a matter of exposure and familiarity than bell-curve positioning. If any backend engineer with 1 year of ORM experience had spent that year instead becoming familiar with SQL, the speed bump would be practically nil.

Re: Ban 1+N in Django

#30

Post by rowanseymour »

I would love to see this become something that can be toggled as a setting in core. The only time it's useful is opening up a shell to debug. The rest of the time, it's just making it really hard to find missing prefetches or `.only()` and `.defer()` calls that are too limited. It's never a good feature for a production site running at any kind of scale.
Post reply on HN