In our services I implemented a per request query counter that gently warns you if you exceed a max query count. Use case: identifying when it might make sense to use DataLoader. Note: doing this isn’t always worth it.
Ban 1+N in Django
81–90 of 153 posts
Re: Ban 1+N in Django
#82In my experience, the far more pernicious way to get N+1 is serializers (Django REST)
Ugh yes, nested serializers in DRF are always fun.
Almost makes you want to go noSQL.
Re: Ban 1+N in Django
#83sentry.io is pretty good at catching N+1 queries.
Sentry has a lot of GDPR problems, it's not easy to set it up so you don't leak PII in e.g. traces sent to it.
Re: Ban 1+N in Django
#84sentry.io is pretty good at catching N+1 queries.
Sentry has a lot of GDPR problems, it's not easy to set it up so you don't leak PII in e.g. traces sent to it.
Correct. Sentry on web and Firebase on mobile app are often storing troves of Personal Data captured by developers.
Neither is meant for PII/Personal Data processing and are huge compliance risks.
Re: Ban 1+N in Django
#85TIL:
> With something so innocent as an attribute access making an SQL query, it’s much easier to miss it.
I like Python as much as the next person, but this is highly irresponsible design decision.
Starting to appreciate Scala's IO effects even more.
Re: Ban 1+N in Django
#86In our services I implemented a per request query counter that gently warns you if you exceed a max query count. Use case: identifying when it might make sense to use DataLoader. Note: doing this isn’t always worth it.
I like this idea. I would just log the query count at the end of each request without any pre-defined limits. IFF you see performance drop, then you can always investigate and see if anything is triggering greater than expected calls.
Nifty idea, and probably not too hard to implement.
Re: Ban 1+N in Django
#87Shameless plug - I ran into this while developing REST interfaces with Django and built django-auto-prefetching: https://github.com/GeeWee/django-auto-prefetching
It essentially travels your DRF serializer tree and builds an auto-prefetched query automatically without you needing to do any work.
Back when I still worked actively on it, I wanted to monkey-patch models to track whether or not n+1 was happening, and if it was, automatically do pre-fetching, so instead of an n+1 problem you'd end up with just a "3-4 queries when it could've been 1" problem - which is much more palatable. Never got around to that part though.
Re: Ban 1+N in Django
#88This is why I always advocated against ORMs. It’s so easy to fall into traps like this without even knowing it, and while you can work around it in some ORMs it is not obvious. Writing SQL is not that hard, and mapping the results to a type isn’t that hard either. So with an ORM you might end up saving several hours of work up front for lots of pain later.
An ORM is about more than mapping results to types, in Djangos case you get a powerful DDL generator, migration management, constraint validation when saving, DB portability, ...
Re: Ban 1+N in Django
#89TIL: > With something so innocent as an attribute access making an SQL query, it’s much easier to miss it. I like Python as much as the next person, but this is highly irresponsible design decision. Starting to appreciate Scala's IO effects even more.
This is not a Python problem, it's a design decision of Django's ORM specifically.
Re: Ban 1+N in Django
#90This is why I always advocated against ORMs. It’s so easy to fall into traps like this without even knowing it, and while you can work around it in some ORMs it is not obvious. Writing SQL is not that hard, and mapping the results to a type isn’t that hard either. So with an ORM you might end up saving several hours of work up front for lots of pain later.
An ORM is about more than mapping results to types, in Djangos case you get a powerful DDL generator, migration management, constraint validation when saving, DB portability, ...
Automated migration management is probably the most important part of Django's ORM.