Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Django 3.1

djangoproject.com

Post reply on HN
Print view View original

191–200 of 209 posts

Re: Django 3.1

#191

Post by georgewalker355 »

mgraczyk wrote: post #166
btown wrote: post #147

Earlier quoted context omitted.

As an FYI: Using a gevent monkey patch has been a way to get async Django for years. Overhead is inefficient in CPU cycles and you need to stay away from doing CPU bound things like Numpy manipulations, but for an app server that’s bound by external API call latency, it practically gives infinite concurrency compared to a thread-per-request model. And no need to worry about event queues. You can feel free to synchron…

I'd add that crucially, DB ORM operations just work with gevent. It will be a while before async database operations are supported natively by Django. For me that is a complete blocker.

> DB ORM operations just work with gevent

My impression was that django+gevent requires some care for the db part. The psycopg2 docs state, for instance

"Psycopg connections are not green thread safe and can’t be used concurrently by different green threads." [1,2].

In addition, gevent cannot monkey patch psycopg2 code because it is C and not python. This is handled by calling psycopg2.extensions.set_wait_callback() [1,3,4]

I just now realize that you're probably referring to making the ORM itself async capable which is on the roadmap https://code.djangoproject.com/wiki/AsyncProject in which case I totally agree.

[1] https://www.psycopg.org/docs/advanced.html#support-for-corou... [2] https://stackoverflow.com/questions/12650048/how-can-i-pool-... [3] https://github.com/gevent/gevent/blob/master/examples/psycop... [4] https://github.com/psycopg/psycogreen

Re: Django 3.1

#193

Post by yebyen »

avremel wrote: post #170
yebyen wrote: post #151

Earlier quoted context omitted.

This is a hot-take from Aaron Patterson, Rails and Ruby core team dev, in his keynote this year where he addressed a very similar idea on a perhaps related query generation topic... https://youtu.be/9JEEabL90AA?t=1360 To give you the tl;dr (he goes through profiling and a great deal of data to help show what a core dev needs to do in order to help us solve this one specific case, and...) Aaron comes to the conclusion…

I would be very hesitant to turn an ORM into a "smart" SQL generator. Depending on the type/distribution of data, there are sometimes very different paths to optimizing a query. The ORM as a "stupid" SQL generator (straightforward mapper) is a great way to allow for the flexibility to control how a query is generated. The DB engine might be the place for those sort of improvements.

In this case the DB engine was not the place for those improvements, as the majority of the extra time was being spent in the "Compile" method which merely composes SQL. If you're worried about turning the ORM into a smart sql generator, having second thoughts about that, in Rails and in ActiveRecord, I'd have to say that ship has already sailed. The ORM is very smart and usually generates pretty good SQL for you. It's never a bad idea to review what it generated and double check based on your own DB knowledge, which might always be better informed than a robot's, but...

Early in the profiling part of the talk Aaron shows how he nearly got hoodwinked into thinking that differences in the query itself were what was causing the slowdown, and while in the end two different queries are still generated by the two different versions of ORM code, the bulk of the performance capture is reclaimed without any impact from the query difference, at least a ~30% performance discrepancy comes solely from the object side of the equation, in Ruby.

(Is it a string or integer parameter? That might make a difference in the query performance... is it one bound array parameter, or one parameter binding per array element? That could only make a difference on the Ruby side, as bound SQL params are always mapped into a query as individual values, at least in this example. These factors are all in play.)

It's a long talk but it's really interesting, (I set a time index in the link to get you past the most frivolous and off-topic parts, which you usually find in a tenderlove talk... that part is not for everyone) I think this talk probably has something for everyone, even if you're not a Rails dev.

Re: Django 3.1

#194

Post by andrewingram »

formerly_proven wrote: post #118
SahAssar wrote: post #111

Earlier quoted context omitted.

I've been working on a project using schema.org objects (100's of types of objects) in postgres in a graph/tree like structure. While structured the objects are not consistent enough for me to want separate schemas for them. Using JSONb in postgres allows me to index separate fields within the JSON while still having all the objects in a single graph. If I couldn't use JSONb with indexes (and triggers for custom fore…

I used to believe in these database antipatterns (e.g. avoid EAV, avoid attribute tables, avoid denormalization, avoid overnormalization, avoid inheritance, avoid wide tables, only use synthetic keys, only use natural keys, never use autoincrement, always use autoincrement), but these days I'm almost certain all of them are wrong, because they are independent of business domain needs. Business domains have antipatter…

Not even just user-defined fields, any time where the fields need to be defined as data values rather than part of your concrete data model. Product attributes on a big multi-category e-commerce site being another example. You could just stick them in JSON, but it's still essentially EAV, just with a different storage mechanism.

Another advantage of EAV as a pattern, is it makes it straightforward to add metadata to values, like "where did this value come from?". For systems with audit trails, just having keys and values isn't enough.

EAV wouldn't be the first tool I'd jump to -- I've definitely seen it go wrong -- but it feels like it's going too far to call it an anti-pattern.

Re: Django 3.1

#195

Post by andrewingram »

konschubert wrote: post #4

Slightly OT: How does HN feel about the recent craze to make web python asynchronous? To me, the performance gains are dubious in many cases and the complexity overhead of handling cooperative multitasking just seems like a step back for a language like python.

If I look at the database queries on the vast majority of pages in a typical Django project, I see a big list of operations being executed sequentially that could actually be done in parallel.

Additionally, (this is my pet use case) if you implement a GraphQL server on top of Django (using one of the many libraries), you tend to get subpar performance because GraphQL lends itself really well to parallelised data fetching, which is hard to take advantage of at the moment.

Re: Django 3.1

#196

Post by andrewingram »

Daishiman wrote: post #178
doteka wrote: post #161

Earlier quoted context omitted.

While a good point, I would say the problem is caused by ORMs existing. Somehow somewhere we decided that a person who thinks SQL is too difficult should be using a database.

Nobody who builds an ORM thinks that. An ORM saves experienced SQL users from having to write boilerplate SQL and hack on their own garbage ORM, which is what any large project ends up doing, attempting to compose queries and filters in vain. I have never ever heard of ORMs as an argument to avoid learning SQL, and AFAIK no author of well-known ORMs holds that opinion.

Yeah, my SQL is admittedly rusty due to not hand-writing many queries these days, but my motivation for using an ORM isn't "I can't write SQL", it's that I don't want to be engaging in string building when there are better abstractions available for supporting this kind of operational composition.

The problem with ORMs, as I see it, is that their abstraction is typically too high level and rarely offers you an intermediate layer to let you work around the leaks.

So you often have:

  ORM -> SQL
Which is implemented as:

  ORM -> private query-builder API -> SQL
When what I want is:

  ORM -> public query-builder API -> public SQL-like abstraction -> SQL

Re: Django 3.1

#197

Post by doteka »

Daishiman wrote: post #178
doteka wrote: post #161

Earlier quoted context omitted.

While a good point, I would say the problem is caused by ORMs existing. Somehow somewhere we decided that a person who thinks SQL is too difficult should be using a database.

Nobody who builds an ORM thinks that. An ORM saves experienced SQL users from having to write boilerplate SQL and hack on their own garbage ORM, which is what any large project ends up doing, attempting to compose queries and filters in vain. I have never ever heard of ORMs as an argument to avoid learning SQL, and AFAIK no author of well-known ORMs holds that opinion.

Anecdata: in every single company I worked, tiny startups and giant corporations alike, “not everyone knows SQL well” was exactly the argument used to justify the ORM. Every single time, the people who do not know SQL, unsurprisingly, proceeded to write pathological N+1 queries.

I don’t mind much - I get to look like a hero to my manager making their queries orders of magnitude faster. But if someone can’t be bothered to write SQL, they for sure will not bother looking up ways to hint the ORM.

Re: Django 3.1

#198

Post by WD-42 »

zelphirkalt wrote: post #174
WD-42 wrote: post #156

Earlier quoted context omitted.

I'm sorry, but this is just programming. You have two choices: 1. Never upgrade and keep everything stable but go through hell when you inevitably must update something or 2: keep up with the latest and go through some minor pain with every release. It's a lot like taking care of a house or car. The world does not stand still, much less programming frameworks.

Hmmm, with well chosen, thought through abstractions, which only presume the minimum they need to presume about their usage, the frequent changes you describe are not a given. The question is, what it is, what frequently needs to change and why no appropriate abstraction has been found to reach stability.

We're talking about a huge web framework here. Your comment might apply to small isolated pieces of code. But Django is built on top of thousands if not millions of abstractions. It's absolutely unreasonable to think that it won't change.

Re: Django 3.1

#199

Post by bredren »

heliodor wrote: post #192
samirsd wrote: post #184

can the async functionality replace celery?

No. Celery can execute tasks after your view returns. Async can't do that.

Would you please elaborate? If all of a task is encapsulated in a django async view, would this not be the same thing?

Re: Django 3.1

#200

Post by bredren »

whouweling wrote: post #9
konschubert wrote: post #4

Slightly OT: How does HN feel about the recent craze to make web python asynchronous? To me, the performance gains are dubious in many cases and the complexity overhead of handling cooperative multitasking just seems like a step back for a language like python.

Well, if you have external API calls in your Django app and you are running sync (which I would absolutely advice, with running async it is really easy to get an unpredictable performance which is sometimes hard to track down) having the ability to run some views async is really crucial. Otherwise your application might me humming along smoothly at some point and coming to a sudden complete standstill or performance…

I've been looking at whether this would be appropriate for something like server-side mixpanel event tracking. Or for sending transactional emails or text notifications using a 3rd party service like mailgun or twilio.

From what I can tell it is not intended for that purpose, and outright will not work.

Post reply on HN