Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

12 requests per second: A realistic look at Python web frameworks

suade.org

Post reply on HN
Print view View original

161–170 of 239 posts

Re: 12 requests per second: A realistic look at Python web frameworks

#161

Post by rkangel »

cryptos wrote: post #58

Why Python at all? About 10 years ago I liked Python a lot (and still like it in principle) and felt very productive compared to, say, Java. Java was full of inconvenience, XML, bloated frameworks and all that. But today you can use Kotlin, that is in my opinion even nicer than Python, with performant frameworks (e. g. Quarkus or Ktor) on the super fast JVM. I don't want to start a language war, but maybe Python is n…

I agree - if you have any chance at the slightest volume (and based on this we're talking 100 concurrent users, not a million) then you are handicapping yourself if you're stuck on a single thread.

Personally I would suggest Elixir (and the Phoenix web framework). It's fully 'parallel by default', will happily saturate all your cores and serve tens of thousands at the same time. The language is simple, the framework is good, although obviously not as many people know Elixir as Python.

Re: 12 requests per second: A realistic look at Python web frameworks

#162

Post by gregmac »

stevenjohns wrote: post #39
imperio59 wrote: post #10

My experience doing perf optimizations in real world systems with many many people writing code to the same app is a lot of inefficiencies happen due to over fetching data, inefficiencies caused by naively using the ORM without understanding the underlying cost of the query, and lack of actual profiling to find where the actual bottlenecks are (usually people writing dumb code without realizing it's expensive). Sure,…

There is lots of truth to this. Some ORMs like Django perform joins in very unsuspecting ways. A simple example is, say, foreign keys. Trying to access the foreign key of an object by doing `book.user.id` does an additional query for the user table to get the ID. It's less known that the id is immediately available by just doing `book.user_id` instead. I've spent time optimising things like text searches down from 20…

The most insidious part about misusing ORMs is it's often not visible for a while. Modern DBMSs on modern hardware are crazy fast, so when you have only a few tens or hundreds of thousand rows in your table, those inefficient and pointless ORM queries are just not noticeable because you still get sub-second response times. As your database grows, the site begins to gets slower and slower, but it's hard to distinguish between the real problem and "I guess we're just handling more requests per second".

I personally love tools like Miniprofiler [1] for this (though maybe there's something better today, it's been a while since I've worked on that type of thing). It's a constant and accessible way to keep an eye on what goes into each request, and I've caught many of those bad queries before they were problems by using it (eg: "WTF, why did it take 9 queries and 250ms to grab what looks to be a single row from a single table?!").

[1] https://miniprofiler.com/

Re: 12 requests per second: A realistic look at Python web frameworks

#163

Post by BiteCode_dev »

polyrand wrote: post #140

Related to ORMs/queries/performance, I have found the following combination really good: * aiosql[0] to write raw SQL queries and having them available as python functions (discussed in [1]) * asyncpg[2] if you are using Postgres * Map asyncpg/aiosql results to Pydantic[3] models * FastAPI[4] Pydantic models become the "source of truth" inside the app, they are designed as a copy of the DB schema, then functions rece…

I always though it would be a nice alternative to an ORM: having a tool that take a marshmallow/pydantic/whatever model, optionally passing it additional db specific options, then it generates a bunch of sql files you can call with aiosql. The whole things would then let you optionally get the result wrapped in a model if you need to, with ORM like helpers for common CRUD things.

That would have the benefit of the standardized api of an ORM and the flexibility of SQL, without the coupling.

Re: 12 requests per second: A realistic look at Python web frameworks

#164

Post by wenc »

nmfisher wrote: post #30
tnash wrote: post #7

Use of ORMs is often a performance choke point. Raw DB queries are often much, much faster. Almost always, the more you abstract, the worse you perform. It's great as a developer but not so great as a user.

I haven't touched an ORM in over 6 years, but unless they've improved since then, I honestly can't think of a single reason why anyone would choose to use one. They're clunky monstrosities that act only as guard-rails for inexperienced developers. Far better to invest a few days (which is realistically all you need) to improve their SQL skills and/or code-review practices.

> I honestly can't think of a single reason why anyone would choose to use one.

In the case of SQLAlchemy (referenced in the article), many people use it as a database connection abstraction rather than an ORM. It's kinda the equivalent of "ODBC" for certain Python libraries like Pandas.

For instance, in Pandas you can write your dataframe to the database by going `df.to_sql(tblname, sqlachemycon)` where sqlalchemycon is the connection instance to any database that SQLAlchemy supports.

https://pandas.pydata.org/pandas-docs/version/0.23.4/generat...

I use SQLAlchemy for this purpose alone and write straight up SQL. I've never used the ORM parts of SQLAlchemy.

Re: 12 requests per second: A realistic look at Python web frameworks

#165

Post by deergomoo »

arcbyte wrote: post #154
paulmd wrote: post #66

Earlier quoted context omitted.

if that's your bag then you can still totally do that with an ORM, hibernate for example lets you just write a whole query in raw SQL while still getting all the benefits of eliding a bunch of boilerplate field copying, having "active" objects with ORM-level update/transaction management, etc. plus it means you don't have to write all the dumb "select * from books where bookName = :bookName" code that obviously can b…

I have never seen anyone, even the ORM "experts" in my jobs, save time with an ORM over what a junior engineer couldn't do faster with plain SQL. The reason is that sure, for your first 10 basic select queries, the ORM saved you half an hour. Then you got to that complicated join and had to resort to looking up archane syntaxes and prototyping attribute quirks for an hour, when the junior guy got the whole thing writ…

> The reason is that sure, for your first 10 basic select queries, the ORM saved you half an hour

Maybe we work in very different fields, but "basic select queries" makes up 90% of what I need to fetch from the database.

If I'm working on a forum and I want to load user 123, with all their posts, all the awards each post has, and the count of friends the user has, with Eloquent (Laravel's ORM), I could do:

`User::with('posts.awards')->withCount('friends')->findOrFail(123);`

That would return me a User model, with a collection "posts" containing a list of Post models, each with a collection "awards" of Award models, and a field `friends_count` with the number of friends. It would run three queries: one to fetch the user, one to fetch the posts, and one to fetch the awards. Depending on how I have configured my models, I can have things like dates automatically hydrated to DateTime objects.

Compare that to plain SQL queries; I would have to fetch the users, including manually writing the subquery for the friend count. Once I had those users I would then have to fetch the posts, and then again for the awards. If I want them in a hierarchy like the ORM example gives me, I then need to loop through each set of records and manually stitch them together. Not difficult, but super tedious.

Sure a complicated join is better done with as little magic as possible, but Eloquent exposes functions for adding subselects, joins, etc. in a way that just reads like SQL (and maps 1:1 underneath).

Re: 12 requests per second: A realistic look at Python web frameworks

#166

Post by zwily »

m12k wrote: post #122
lacksconfidence wrote: post #114

Earlier quoted context omitted.

No, that is clearly a bug. The ORM already has the value of book.id, that's how it knows how to fetch the right book. Performing extra queries is just poor implementation.

The programmer also already had the value in book.user_id but still chose to ask the ORM to fetch all of .user so they could get .id from there instead. And they might then afterward call .name on it as well, and there would be no further queries, because the ORM has already been asked to fetch all fields of .user - so it might in fact have been sensible to fetch all of .user if so. The query builder cannot know whet…

Fancier ORMs will return a proxy for foo.user that doesn’t issue a query until the user asks for a property on it. And it can return user.id without querying.

Re: 12 requests per second: A realistic look at Python web frameworks

#168

Post by musingsole »

dfgdghdf wrote: post #132

I would question choosing Python for large server projects because the performance ceiling is so low. At least with the "middle tier" performance languages such as Java / C# you are unlikely to require a complete language switch as the project scales.

With cloud and infinite horizontal scaling...hitting that performance ceiling is an indication of a poor design.

Re: 12 requests per second: A realistic look at Python web frameworks

#169

Post by fouc »

ctvo wrote: post #153

It's a bit of a step back in time reading things like this. This is stateless HTTP requests hitting a relational database. How is this dead horse still being beaten? The patterns for load balancing, horizontal scalability, caching in this space well documented. What are we gaining still profiling Django, Flask and Ruby on Rails in 2021.

Does this mean you work with a stateful websockets setup? What stack?
Post reply on HN