51–60 of 239 posts
When you start hitting bottlenecks in your python web framework its probably time to switch to a faster language, not another framework in python. You're probably done rapid prototyping by this point anyway.
When you start hitting bottlenecks in your python web framework its probably time to switch to a faster language, not another framework in python. You're probably done rapid prototyping by this point anyway.
Earlier quoted context omitted.
I increasingly lean towards plain SQL over ORMs. It requires greater familiarity with SQL but I prefer that over greater familiarity with ORM-specific syntax that doesn’t translate across frameworks or languages. In addition, you can prototype new queries and profile existing queries in the database and copy-paste directly into your code.
In the java-world there are libraries like JDBI, which makes it possible to write interfaces with SQL-annotations and have serialization, connection setup etc. done for you: public interface UserDao { @SqlUpdate("CREATE TABLE user (id INTEGER PRIMARY KEY, name VARCHAR)") void createTable(); @SqlUpdate("INSERT INTO user(id, name) VALUES (?, ?)") void insertPositional(int id, String name); @SqlQuery("SELECT * FROM user…
context
.update(User.USER)
.set(User.USER.NAME, userName)
.where(User.USER.ID.eq(userId))
.execute()Earlier quoted context omitted.
I haven't touched an ORM in over 6 years, but unless they've improved since then, I honestly can't think of a single reason why anyone would choose to use one. They're clunky monstrosities that act only as guard-rails for inexperienced developers. Far better to invest a few days (which is realistically all you need) to improve their SQL skills and/or code-review practices.
I am by no means an experienced developer--but the issue I always seem to have without using an ORM is that there are string literals containing different bits of SQL scattered all throughout my code. In addition to being hard to refactor when the database model changes, I think there is a fairly high performance cost to doing so many string concatenations every time the code runs. Is there a better way to manage the…
The exceptions are SQL elements that cannot be (easily, or at all) parametrized, such as the column lists in SELECT, ORDER BY, GROUP BY, or changing the WHERE "shape" and so on.
In my experience, this tends to be a minority of queries, although an important minority.
----
P.S. If string concatenations are your bottleneck, then your database is screaming fast! The real-life bottlenecks are usually in excessive database round-trips and unoptimized query plans, and are orders-of-magnitude larger.
Please donate. Pypy needs funds - https://opencollective.com/pypy
Pypy doesnt get a fraction of the funding that python does.
Earlier quoted context omitted.
> (usually people writing dumb code without realizing it's expensive) Some years ago, one morning I gave a co-worker a recommendation on how to improve a loop that was unnecessarily hitting database through the Django ORM. He committed the fix that afternoon. Barely an hour later I accidentally reintroduced the exact same slowdown in the exact same loop when adding a different piece of data to it. Soooo yeah, ORMs ca…
That's a nice benefit of using async ORMs (not yet available in django), the db calls are explicit!
In my benchmark testing, SSL appears to be the bottleneck; e.g., Apache vs. Nginx does not really matter. I assume the benchmarks above 10,000 RPS are not using SSL and regular HTTP? How are people doing benchmarks at 10k-100k RPS?
I don't want to start a language war, but maybe Python is not the first choice for their requirements.
Earlier quoted context omitted.
I increasingly lean towards plain SQL over ORMs. It requires greater familiarity with SQL but I prefer that over greater familiarity with ORM-specific syntax that doesn’t translate across frameworks or languages. In addition, you can prototype new queries and profile existing queries in the database and copy-paste directly into your code.
I favor a hybrid approach: use the Django ORM to define models, do migrations, auto generate the admin, etc. but don’t be shy about using the extension points (extra, raw, cursors) to put in an optimized query for a hotspot. You can get pretty far using the ORM but it’s really valuable to be able to be comfortable dropping down for things like reports or bulk processing.
Earlier quoted context omitted.
What makes postgres special in these cases?
I would like to know as well. All I know is that for this specific workload there appears to be a strong correlation between the chosen DB and performance. Depending on the benchmark you're looking at on that site, the top 30-60 results are exclusively postgres. Further, for every framework/language that was tested with both MySQL and postgres (there's quite a few of them), the postgres one always ranks higher.
Perhaps it's because pg had better async drivers?