Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10 000 concurrent real-time connections to Django

github.com

Post reply on HN
Print view View original

11–20 of 40 posts

Re: 10 000 concurrent real-time connections to Django

#11

Post by mYk »

stefantalpalaru wrote: post #10
mYk wrote: post #7

Earlier quoted context omitted.

The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (wi…

Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!

Yes, I'm just referring to C10k because this demo uses a technique originally created to solve C10k.

Reaching 10 000 connections wasn't difficult in this case; it was just a matter of tuning a few system parameters. Exploring the APIs and studying how they can fit together was much more interesting, and sometimes challenging.

Re: 10 000 concurrent real-time connections to Django

#12

Post by vampirechicken »

stefantalpalaru wrote: post #4

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html

Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.

Re: 10 000 concurrent real-time connections to Django

#13

Post by eyepulp »

mYk wrote: post #9
kapilvt wrote: post #5

Earlier quoted context omitted.

how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.

This system doesn't replace Django; it complements it. You could build 95% of an application with the traditional request-response model and add the 5% of real-time featurs with a system similar to my demo. (Of course, given your opinions on Django, I don't recommend you build anything with it.)

Well said =) It's incredibly frustrating to so easily build the entire app with a "traditional" django stack and then be faced with solving for a whole new stack just to avoid wasteful XHR-polling for some simple server-side event driven UI updates.

We've implemented some SSE based solutions lately with gevent & nginx in front of django, and it's been great to keep it all in the django family.

Maybe switching to py3k has some value after all... =)

Re: 10 000 concurrent real-time connections to Django

#14

Post by jaytaylor »

vampirechicken wrote: post #12
stefantalpalaru wrote: post #4

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html

Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.

Django hasn't stopped insisting on that... v1.5 might have pooling, but it's taken them a long time to get connection pooling in.

Re: 10 000 concurrent real-time connections to Django

#17

Post by obviouslygreen »

jaytaylor wrote: post #14
vampirechicken wrote: post #12

Earlier quoted context omitted.

Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.

Django hasn't stopped insisting on that... v1.5 might have pooling, but it's taken them a long time to get connection pooling in.

Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).

Re: 10 000 concurrent real-time connections to Django

#18

Post by jaytaylor »

obviouslygreen wrote: post #17
jaytaylor wrote: post #14

Earlier quoted context omitted.

Django hasn't stopped insisting on that... v1.5 might have pooling, but it's taken them a long time to get connection pooling in.

Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).

Yes, but pgpool isn't always an option. If you use Heroku's Postgres service, you do not have sufficient privileges to run the commands to attach to software like pgpool.

The only way we were able to get connection pooling working in django 1.3/1.4 was to use django-dbpool[0]. It works okay, but is still pretty sketchy compared to the connection pooling libraries available for the JVM like BoneCP or C3P0.

[0] https://github.com/gmcguire/django-db-pool

Re: 10 000 concurrent real-time connections to Django

#19

Post by notdonspaulding »

eyepulp wrote: post #13
mYk wrote: post #9

Earlier quoted context omitted.

This system doesn't replace Django; it complements it. You could build 95% of an application with the traditional request-response model and add the 5% of real-time featurs with a system similar to my demo. (Of course, given your opinions on Django, I don't recommend you build anything with it.)

Well said =) It's incredibly frustrating to so easily build the entire app with a "traditional" django stack and then be faced with solving for a whole new stack just to avoid wasteful XHR-polling for some simple server-side event driven UI updates. We've implemented some SSE based solutions lately with gevent & nginx in front of django, and it's been great to keep it all in the django family. Maybe switching to py3k…

I concur.

Re: 10 000 concurrent real-time connections to Django

#20

Post by talmirza »

jaytaylor wrote: post #18
obviouslygreen wrote: post #17

Earlier quoted context omitted.

Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).

Yes, but pgpool isn't always an option. If you use Heroku's Postgres service, you do not have sufficient privileges to run the commands to attach to software like pgpool. The only way we were able to get connection pooling working in django 1.3/1.4 was to use django-dbpool[0]. It works okay, but is still pretty sketchy compared to the connection pooling libraries available for the JVM like BoneCP or C3P0. [0] https:/…

You can still us pg_pool with heroku.
Post reply on HN