Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10 000 concurrent real-time connections to Django

github.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: 10 000 concurrent real-time connections to Django

#31

Post by stefantalpalaru »

obviouslygreen wrote: post #17
jaytaylor wrote: post #14

Earlier quoted context omitted.

Django hasn't stopped insisting on that... v1.5 might have pooling, but it's taken them a long time to get connection pooling in.

Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).

Yes, connecting repeatedly to pgpool is less expensive than doing it directly to postgres. Not having to connect repeatedly to anything is even better.

Re: 10 000 concurrent real-time connections to Django

#32

Post by StavrosK »

stefantalpalaru wrote: post #30
StavrosK wrote: post #28

Earlier quoted context omitted.

By "implemented", do you mean "contributed to trunk", or "used it in my application"? If the former, thanks, if the latter, I was under the impression that it was on by default in 1.5?

Looks like it will be available in the next (1.6?) release[1]. [1] https://docs.djangoproject.com/en/dev/ref/databases/#persist...

Oh right, 1.5 and earlier. I misread that, thank you.

Re: 10 000 concurrent real-time connections to Django

#33

Post by bayesianhorse »

kapilvt wrote: post #5
LeafStorm wrote: post #3

Earlier quoted context omitted.

Because it's a popular, well-written framework with a huge volume of existing code, and integrating WebSockets into Django means you don't have to maintain two codebases if you want to have a realtime Django app. Also, it's fun.

how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.

In my opinion there is nothing wrong with doing simple Django ORM calls even with a tornado/gevent websocket. Async purists will cry foul, but if all you want is a realtime feature, you can block for a few milliseconds without getting into trouble.

In the long run they might build tulip-awareness into the ORM.

For that matter I start Tornado servers with management commands just for convenience...

Re: 10 000 concurrent real-time connections to Django

#34

Post by taude »

vampirechicken wrote: post #12
stefantalpalaru wrote: post #4

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html

Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.

Wow. I'd just have assumed that connection pooling was default in all these modern web frameworks. I guess this is one area where Enterprise means you get what you pay for? Example: ODBC connection pooling with ASP pages back in the late '90s.

Re: 10 000 concurrent real-time connections to Django

#35

Post by twism »

bayesianhorse wrote: post #33
kapilvt wrote: post #5

Earlier quoted context omitted.

how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.

In my opinion there is nothing wrong with doing simple Django ORM calls even with a tornado/gevent websocket. Async purists will cry foul, but if all you want is a realtime feature, you can block for a few milliseconds without getting into trouble. In the long run they might build tulip-awareness into the ORM. For that matter I start Tornado servers with management commands just for convenience...

Or you could do this ... https://gist.github.com/anonymous/5190528

This blocks in a separate thread leaving the event loop to handle other tasks.

Re: 10 000 concurrent real-time connections to Django

#36

Post by the_mitsuhiko »

vampirechicken wrote: post #12
stefantalpalaru wrote: post #4

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html

Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.

Connection pooling in postgres can actually be counter productive with high concurrency. Postgres scales very badly to high number of connections.

Re: 10 000 concurrent real-time connections to Django

#38

Post by raverbashing »

stefantalpalaru wrote: post #29
raverbashing wrote: post #27

Earlier quoted context omitted.

Well, about PgBouncer, I believe this tells more of PostgreSQL than of Django + psql libraries (After all it will still open and close connections for Django but keeps them open for psql) Now, for serving 10k connections with template libraries and middlewares and ORM, out of the box? Serving dynamic content for each connection? Impossible =) Not without some kind of caching (but you can do that with the help of a mi…

It's not PostgreSQL's fault that Django closes the connection after each request explicitly (see close_connection() for what was done before 1.6): [1] https://github.com/django/django/blob/master/django/db/__ini...

I understand that, and I'm not saying that there aren't several issues in Django (but some are getting better)

What I'm saying is that with PgBouncer you have Django PgBouncer Postgresql, right?

So what PgBouncer is doing (dealing with several open/close connections) maybe could be done better inside PostgreSQL

Re: 10 000 concurrent real-time connections to Django

#40

Post by hoov »

est wrote: post #16

For Python2.X, I think gevent.wsgi.WSGIServer could handle C10k or more, no?

In my experience -- yes. I've pushed it further than that.

In practice, debugging is an unpleasant experience. If you're doing anything slightly out of the ordinary, you should take care.

Post reply on HN