Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

12 requests per second: A realistic look at Python web frameworks

suade.org

Post reply on HN
Print view View original

221–230 of 239 posts

Re: 12 requests per second: A realistic look at Python web frameworks

#221

Post by polyrand »

BiteCode_dev wrote: post #163
polyrand wrote: post #140

Related to ORMs/queries/performance, I have found the following combination really good: * aiosql[0] to write raw SQL queries and having them available as python functions (discussed in [1]) * asyncpg[2] if you are using Postgres * Map asyncpg/aiosql results to Pydantic[3] models * FastAPI[4] Pydantic models become the "source of truth" inside the app, they are designed as a copy of the DB schema, then functions rece…

I always though it would be a nice alternative to an ORM: having a tool that take a marshmallow/pydantic/whatever model, optionally passing it additional db specific options, then it generates a bunch of sql files you can call with aiosql. The whole things would then let you optionally get the result wrapped in a model if you need to, with ORM like helpers for common CRUD things. That would have the benefit of the st…

Absolutely! I think it should be doable, FastAPI and other libraries do heavy Pydantic model inspection, and some do automatic code generation. I should explore this a bit more.

Re: 12 requests per second: A realistic look at Python web frameworks

#222

Post by polyrand »

fokinsean wrote: post #186
polyrand wrote: post #140

Related to ORMs/queries/performance, I have found the following combination really good: * aiosql[0] to write raw SQL queries and having them available as python functions (discussed in [1]) * asyncpg[2] if you are using Postgres * Map asyncpg/aiosql results to Pydantic[3] models * FastAPI[4] Pydantic models become the "source of truth" inside the app, they are designed as a copy of the DB schema, then functions rece…

This sounds like a really cool stack. I'm testing the waters with FastAPI and SQLAlchemy right now, but SQLAlchmey feels like it just gets in the way. Do you have an example project which uses all of these I could look at?

I was writing a cookiecutter template for this kind of stack but never finished it. After using similar stacks for a couple of projects I am going to get back to it soon with many improvements and publish it.

Re: 12 requests per second: A realistic look at Python web frameworks

#223

Post by polyrand »

jordic wrote: post #181
polyrand wrote: post #140

Related to ORMs/queries/performance, I have found the following combination really good: * aiosql[0] to write raw SQL queries and having them available as python functions (discussed in [1]) * asyncpg[2] if you are using Postgres * Map asyncpg/aiosql results to Pydantic[3] models * FastAPI[4] Pydantic models become the "source of truth" inside the app, they are designed as a copy of the DB schema, then functions rece…

We do something like this, also we took some inspiration from hashura. Asyncpg it's so fast an ergonomic.

That sounds great! I’ve always kept an eye on how Hasura does things. Do you have any open projects using this stack?

Re: 12 requests per second: A realistic look at Python web frameworks

#224

Post by pas »

hendry wrote: post #113

I always assumed Python could scale because of Reddit: https://github.com/reddit-archive/reddit Not quite sure if their current site's code is opensource... anyone know?

Facebook uses/used PHP, that scales too. If you make it stateless and use it only as a glue. Dropbox uses Python too. Eve Online used Stackless Python.

Of course Reddit/FB/EVE/Dropbox shards everything, there's no global state to manage via Python. The state lives in the data store layer.

And for that there are these monstrous/elegant things like Vitess, that YouTube used (uses?): https://vitess.io/docs/overview/architecture/ which is basically a sharding/routing layer on top of independent MySQL instances.

Re: 12 requests per second: A realistic look at Python web frameworks

#225

Post by alex-lx »

lukaseder wrote: post #211
alex-lx wrote: post #91

Earlier quoted context omitted.

jOOQ and its DSL is good, however IMO it's more readable using raw SQL (by using `context.fetchInto` and its variants) than to using DSL when deal with complex query.

Why not just create views and query those with jOOQ, then?

hi, @lukaseder, Thank you for creating jOOQ. I don't have much experience on views, and some called `Best Practice` forbid to use that because it's hard to maintain. What do you think about that.

Re: 12 requests per second: A realistic look at Python web frameworks

#226

Post by Izkata »

ehutch79 wrote: post #150
Izkata wrote: post #148

Earlier quoted context omitted.

> It's not a bug, it is the same value. Not necessarily, it can be overridden: "id" is only the default for models that haven't explicitly been given a field with the "primary_key" kwarg (common on legacy tables where the primary key column might for example be "user_id"). The alias guaranteed to be the same value is ".pk", and I'm not sure what django does if you try to create a column named "pk" that isn't the prim…

If your tables don't use id, or are doing something strange, YOU wrote that code. You should know it's going to be a thing you need to deal with, because YOU did something unusual.

Note that I gave the example of "legacy" tables. On long-lasting codebases there's a high likelihood that "you" didn't write that code and aren't aware of everything Django's doing under the hood.

Re: 12 requests per second: A realistic look at Python web frameworks

#227

Post by adsharma »

polyrand wrote: post #140

Related to ORMs/queries/performance, I have found the following combination really good: * aiosql[0] to write raw SQL queries and having them available as python functions (discussed in [1]) * asyncpg[2] if you are using Postgres * Map asyncpg/aiosql results to Pydantic[3] models * FastAPI[4] Pydantic models become the "source of truth" inside the app, they are designed as a copy of the DB schema, then functions rece…

It would be cool to teach the python static type checkers such as pyright and mypy to do the same validation as pydantic, marshmallow.

Then you could use dataclasses and map them to the database via sqlalchemy.

https://github.com/adsharma/dataclasses-sql

Couple of other techniques to speedup python:

* Transpile python to another language (py2many) * Compile a large graphql like query to a single query plan in python which can be accelerated. (Fquery)

Both projects on my github.

Re: 12 requests per second: A realistic look at Python web frameworks

#228

Post by arcturus17 »

nodamage wrote: post #205
arcturus17 wrote: post #124

Earlier quoted context omitted.

It's still 8th in the composite benchmark. And the criticism you're leveling would affect the entire benchmark design, rather than a particular framework score, no? [1] https://www.techempower.com/benchmarks/#section=data-r20&hw=...

> And the criticism you're leveling would affect the entire benchmark design, rather than a particular framework score, no? Indeed. The problem is that many of the scores in the top 100 are really misleading because no one building a web app would implement things that way. There is still some value in the lower down benchmarks but you have to basically read the underlying source to determine if the implementation is…

Yea I've seen that they do provide ORM benchmarks in the detailed tests. I still find it very impressive - as a full-featured framework it ranks well above the rest of its peers, despite the ORM.

I am writing Django backends nowadays, but like I've mentioned elsewhere, I love its full-batteries approach and maturity, but I'm not very happy about it's meager async capabilities, and I think I'd prefer something with strong typing...

I've always been put off by Microsoft's lock-in but it seems that's changed so at least I'd put the .net as a contender for side project in the near future.

Re: 12 requests per second: A realistic look at Python web frameworks

#229

Post by mixmastamyk »

dfgdghdf wrote: post #207
musingsole wrote: post #204

Earlier quoted context omitted.

If you've made it impossible or just difficult to scale horizontally in the server world, yeah, you've got a bad design.

Sorry to keep coming back, I don't want to start a flame war here. Difficulty of horizontal scaling is a property of two things: 1. how your solution is designed (your point, and I agree this is often done poorly) 2. the problem / work-load you are trying to scale (my point). It might be that your problem does not "shard" very easily. You cannot fix this with solution architecture, at least not easily. Horizontal sca…

There are lots of ways to speed python if need be, and you can recover much of the work already done. Cython is another.

Re: 12 requests per second: A realistic look at Python web frameworks

#230

Post by akra »

Ambix wrote: post #182
hgretg3443 wrote: post #65

C#/ASP.NET is the fastest web framework now: https://www.techempower.com/benchmarks/#section=test&runid=8... 7.000.000 requests per second Even GO can only achieve 4.500.000 million requests per secnod being a low-level language, in opposite to high-level C#.

Why do enyone needs framework to print zillion Hello Worlds to the client? It's more interesting to see results of high-load DB tests, for example: https://www.techempower.com/benchmarks/#section=data-r20&hw=...

Depends on your use case. I've written quite a few backends that didn't use a database and there are a number of cases for the pure HTTP benchmark. Pass through proxies with injected behaviour, in memory key-value data storage APIs, memory mapped files, etc. IMO not every shop offloads the state to its DB. Once you add the database you are really testing the DB driver - and that adds a lot more variance to the test. Maybe its just the DB driver for that particular DB?

A fast "Hello World" benchmark implies that the HTTP/transport layers of the framework are very fast. That's your base and the lower bound to your best performance potential. As a real world example if .NET ASP NET Core has the best request/response benchmark and its better than say nginx (a popular reverse proxy) it might be better to have all your gateways using a reverse proxy implemented with that as its base instead. Over your whole network depending on your scale that could be a big cost and latency saving measure. I wouldn't be surprised if Microsoft or it's community have started writing one.

Post reply on HN