Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban 1+N in Django

suor.github.io

Post reply on HN
Print view View original

121–130 of 153 posts

Re: Ban 1+N in Django

#121

Post by stavros »

har777 wrote: post #75

Self plug: Checkout https://github.com/har777/pellet to easily find and fix django N+1 issues. I usually add it to existing integration tests so that they raise exceptions on N+1. If test coverage is low then I would suggest sending the N+1 metrics to something like datadog. That way your users using the product will reveal all the N+1 issues on your monitoring solution. EDIT: I should add a screenshot to the README…

This is great!

I also use https://pypi.org/project/django-zen-queries/, but Pellet might be better (Zen queries doesn't let you run any queries, which sometimes doesn't work).

Here's a screenshot for you:

https://imgz.org/i8XkiK2R.png

Re: Ban 1+N in Django

#122

Post by stavros »

jonatron wrote: post #8

See also django-zen-queries https://github.com/dabapps/django-zen-queries , which can make it impossible for changes to a template to trigger queries.

I like Zen queries, but sometimes you just have to make queries in templates (e.g. when you want to check the user on the request object), or sometimes it's just convenient and there's nothing wrong with it (e.g. when you want to check the user on the request object).

Zen queries makes that use case impossible, sadly.

Re: Ban 1+N in Django

#123

Post by orf »

cnity wrote: post #120
orf wrote: post #119

Earlier quoted context omitted.

Whilst shallowly funny the sarcasm actually shows a lack of understanding of the problem. The problem can be described as this: given an arbitrary point in a program, how can you infer what data is required at a future point without executing code between those two points? I’d love to see you go make a solution to this.

The point is that there is no "generic" solution. The abstraction of the ORM has created the illusion that you don't need to worry about this, the ORM will handle it. N+1 examples like iterating over all the publications from an author arise because the goal is to not have to concern yourself with the fundamentals of data fetching.

I’m not sure that position lives up to reality. It’s easy to avoid N+1 queries in Django, there are lots of ways to reduce that to 1 joined query or 2 in the case of a m2m. The issue here isn’t that “orms are bad”, the issue here is that “knowing how data flows through your system is hard”.

You’d have exactly the same issue as you’d have if you’d call “get_author_publications” O(n) times in some nested call stack. Except getting out of it wouldn’t be as easy as calling “select_related(…)” - you’d instead just end up re-inventing an ORM.

Re: Ban 1+N in Django

#124

Post by irjustin »

cronopios wrote: post #95
irjustin wrote: post #32

Rails has Bullet[0] to help identify and warn you against N+1 Does Django have anything active? Quick search revealed nplusone[1] but its been dead since 2018. [0] https://github.com/flyerhzm/bullet [1] https://github.com/jmcarp/nplusone

If I understand it correctly, that's what [django-seal]( https://github.com/charettes/django-seal ) does.

definitely seems like a good candidate.

Re: Ban 1+N in Django

#125

Post by jorl17 »

paulddraper wrote: post #33

In my experience, the far more pernicious way to get N+1 is serializers (Django REST)

A couple of years ago I wrote a set of utility classes (which inherit from Serializer and ViewSet) which "solve" this problem by inspecting serializers at the beginning of the request and figuring out what to pass to `select_related` and `prefetch_related` "automatically". It supports nested serializers, N-N, etc. Also lets you "help it" by saying "assume attribute X of a serializer accesses fields X,Y,Z of its object", for more "sophisticated" cases.

It's a very messy piece of code but it has survived many projects since I first wrote it 6 years ago. The day I enabled it at a previous job, we reduced a page load from 20s to 800ms or so just with it.

Re: Ban 1+N in Django

#126

Post by cnity »

orf wrote: post #123
cnity wrote: post #120

Earlier quoted context omitted.

The point is that there is no "generic" solution. The abstraction of the ORM has created the illusion that you don't need to worry about this, the ORM will handle it. N+1 examples like iterating over all the publications from an author arise because the goal is to not have to concern yourself with the fundamentals of data fetching.

I’m not sure that position lives up to reality. It’s easy to avoid N+1 queries in Django, there are lots of ways to reduce that to 1 joined query or 2 in the case of a m2m. The issue here isn’t that “orms are bad”, the issue here is that “knowing how data flows through your system is hard”. You’d have exactly the same issue as you’d have if you’d call “get_author_publications” O(n) times in some nested call stack. Ex…

> Except getting out of it wouldn’t be as easy as calling “select_related(…)” - you’d instead just end up re-inventing an ORM.

This could be where we differ. I wouldn't propose that at all, I'd propose writing a query more appropriate for whatever part of the code contains the iteration.

Re: Ban 1+N in Django

#127

Post by aforwardslash »

rtpg wrote: post #93
aforwardslash wrote: post #76

Earlier quoted context omitted.

... Or by people that actually understand the impedance mismatch between objects and data (quick django example - request data and models are different and not easily interchangeable). Or people that require good caching implementations. Or people that actually design database systems schema-first. Or peoplw that rely on advanced usage that isnt always easy to perform in orm's. The list goes on.

Django's ORM is much better than most, to the point where Django considers that "I can't model this query with the ORM" to be a bug. There are of course some mismatches, but it's pretty hard to have a query that is not at all modelable, and Django's ORM is ... fairly predictable (I have some gripes about how obvious or not joins are but it's subjective).

There is a subtle difference between "can't be done" or "I'll spend an afternoon digging through documentations, code & examples to implement this".

Quick obvious example - use views for data retrieval (that may contain more fields than the actual model) and tables for data insertion.

Re: Ban 1+N in Django

#128

Post by JeremyNT »

nickjj wrote: post #7

There is a case where having N+1 queries are beneficial. In Rails terms, it's when you perform Russian doll caching, but you can do this in any framework. The idea is you can cache a specific X thing which might make a query to an associated Y thing. A textbook N+1 query case (ie. a list of posts (X) that get the author's name (Y)). If you render the view without any cache with 10 things then you'd perform 20 queries…

> If you render the view without any cache with 10 things then you'd perform 20 queries but after the cache is warm you'd perform 0 queries. If item 5's Y gets updated then you only need to bust the cache for item 5 and query only item 5's Y association. Performing a preloaded query to get all X 10 things with their Y associated things could be an expensive query.

The technique you describe makes sense, but how often is that preload query actually that expensive? Usually if I see an expensive query in Rails, it's because either 1) the indexes are missing for the joins or 2) you're instantiating extra full-blown objects when only some tiny amount of data of a primitive type is required.

This kind of stuff is a good opportunity to just write the query directly that gets the data you really do need, rather than relying on the ORM and its overhead.

If the problem is "maybe the user will want to see this, maybe they won't" then in many cases the easiest win is to lazy load the more detailed view with a new query that is only fired on some user interaction.

Re: Ban 1+N in Django

#129

Post by orf »

cnity wrote: post #126
orf wrote: post #123

Earlier quoted context omitted.

I’m not sure that position lives up to reality. It’s easy to avoid N+1 queries in Django, there are lots of ways to reduce that to 1 joined query or 2 in the case of a m2m. The issue here isn’t that “orms are bad”, the issue here is that “knowing how data flows through your system is hard”. You’d have exactly the same issue as you’d have if you’d call “get_author_publications” O(n) times in some nested call stack. Ex…

> Except getting out of it wouldn’t be as easy as calling “select_related(…)” - you’d instead just end up re-inventing an ORM. This could be where we differ. I wouldn't propose that at all, I'd propose writing a query more appropriate for whatever part of the code contains the iteration.

> I'd propose writing a query more appropriate for whatever part of the code contains the iteration.

This is an identical solution to adding “select_related()” in the right places, and thus we loop back to the start of our thread.

Re: Ban 1+N in Django

#130

Post by aforwardslash »

zzzeek wrote: post #116
aforwardslash wrote: post #74

Earlier quoted context omitted.

Nitpick: no, you've written a pure object mapper, that doesnt care about schema relations. This has the practical advantage of being just a data container that can be clearly serialized/deserialized, instead of a model object with a transitive database connection dependency.

I don't know what serialization/deserialization has do with this. Does the object map to database rows and there's code that moves the data back and forth? That's an object relational mapper.

mapping database rows to object structures is an object mapper. An object relational mapper also keeps track of table dependencies (such as related fields).

If you read a row from a database and generate an object whose attributes map the fields in the database and are used to retrieve the values (a data object), that is an object mapper. This means when fetching eg. user.type it will return 1 instead of the data object for the corresponding row on user_type;

If you read a row from a database, exactly like the above, but user.type returns a data object representing the related table row, that's an object relational mapper.

Regarding serialization, why does it matter? Because you need to serialize and de-serialize data objects or models if you're adding cache to eg. a service layer. Also, serialization and de-serialization are quite important when interfacing with eg. external systems - Imagine having an application-wide InvoiceModel that can be transported via REST, GRPC, kafka/json or any other format, and that is database-agnostic.

Post reply on HN