For Python2.X, I think gevent.wsgi.WSGIServer could handle C10k or more, no?
Potentially it is slower than tulip, It does magic stack slicing. also it works only on X86 cpu.
10 000 concurrent real-time connections to Django
21–30 of 40 posts
Re: 10 000 concurrent real-time connections to Django
#22I dabbled in realtime connections for a while now, and this is certainly a sexy option to avoid the fragmentation happening in the Python async world.
Well, I think that was Guido's goal in creating Tulip.
Re: 10 000 concurrent real-time connections to Django
#23Earlier quoted context omitted.
Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).
Yes, but pgpool isn't always an option. If you use Heroku's Postgres service, you do not have sufficient privileges to run the commands to attach to software like pgpool. The only way we were able to get connection pooling working in django 1.3/1.4 was to use django-dbpool[0]. It works okay, but is still pretty sketchy compared to the connection pooling libraries available for the JVM like BoneCP or C3P0. [0] https:/…
Understood, thank you for the clarification.
Re: 10 000 concurrent real-time connections to Django
#24Earlier quoted context omitted.
Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
Yes, I'm just referring to C10k because this demo uses a technique originally created to solve C10k. Reaching 10 000 connections wasn't difficult in this case; it was just a matter of tuning a few system parameters. Exploring the APIs and studying how they can fit together was much more interesting, and sometimes challenging.
Awesome stuff!
Re: 10 000 concurrent real-time connections to Django
#25Earlier quoted context omitted.
Because it's a popular, well-written framework with a huge volume of existing code, and integrating WebSockets into Django means you don't have to maintain two codebases if you want to have a realtime Django app. Also, it's fun.
how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.
In addition to templating, orm and routing there is also highly integrated i18n toolchain/workflow, built-in csrf protection, good session handling, well integrated caching and many more of such 'details'.
All those things by themselves could be considered trivial and could be gotten from many individual libraries - the level of integration and polish you encounter with Django is anything but trivial though and takes real time and effort.
I've recently had to make a choice for a Python based application platform/environment and have chosen Django (again) despite having no use for the ORM and ORM-using contrib modules. Simply because all the other things are there and work together beautifully without me spending any time on code that is not directly related to my goals.
Re: 10 000 concurrent real-time connections to Django
#26Earlier quoted context omitted.
Because it's a popular, well-written framework with a huge volume of existing code, and integrating WebSockets into Django means you don't have to maintain two codebases if you want to have a realtime Django app. Also, it's fun.
how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.
Trivial or not, working within an existing library provides a level of maintainability you just don't get when you toss in something like Tornado to do the tiny bit of your app that requires a persistent connection and websockets. Even if all I got was django's router and views, it's save me the awful headache of having to graft Tornado and Django together, coding under two entirely different library sets.
Re: 10 000 concurrent real-time connections to Django
#27Earlier quoted context omitted.
The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (wi…
Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
Well, about PgBouncer, I believe this tells more of PostgreSQL than of Django + psql libraries
(After all it will still open and close connections for Django but keeps them open for psql)
Now, for serving 10k connections with template libraries and middlewares and ORM, out of the box? Serving dynamic content for each connection? Impossible =)
Not without some kind of caching (but you can do that with the help of a middleware)
Re: 10 000 concurrent real-time connections to Django
#28The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html
The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (wi…
By "implemented", do you mean "contributed to trunk", or "used it in my application"? If the former, thanks, if the latter, I was under the impression that it was on by default in 1.5?
Re: 10 000 concurrent real-time connections to Django
#29Earlier quoted context omitted.
Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
Well, about PgBouncer, I believe this tells more of PostgreSQL than of Django + psql libraries (After all it will still open and close connections for Django but keeps them open for psql) Now, for serving 10k connections with template libraries and middlewares and ORM, out of the box? Serving dynamic content for each connection? Impossible =) Not without some kind of caching (but you can do that with the help of a mi…
It's not PostgreSQL's fault that Django closes the connection after each request explicitly (see close_connection() for what was done before 1.6):
[1] https://github.com/django/django/blob/master/django/db/__ini...
Re: 10 000 concurrent real-time connections to Django
#30Earlier quoted context omitted.
The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (wi…
By "implemented", do you mean "contributed to trunk", or "used it in my application"? If the former, thanks, if the latter, I was under the impression that it was on by default in 1.5?
Looks like it will be available in the next (1.6?) release[1].
[1] https://docs.djangoproject.com/en/dev/ref/databases/#persist...