Live data from Hacker News

Write Fast Apps Using Async Python 3.6 and Redis

eng.paxos.com

51–60 of 134 posts

Re: Write Fast Apps Using Async Python 3.6 and Redis

#51
post #2

> Paxos.com I'm confused by the relationship between Paxos, the company, and Paxos, the algorithm. Do the authors of Paxos work for Paxos? Edit: https://en.m.wikipedia.org/wiki/Paxos_(computer_science) Ah; both are named for a fictional financial systen

By the way, the author of the original Paxos paper is Leslie Lamport, who currently works at Microsoft Research.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#52

Earlier quoted context omitted.

You normally use something like redis to store the state. If you were going to share state in memory between threads, how would you handle the case where the second request goes to a different server or that the process has restarted? You'd need redis anyway, so you might as well just use it in all cases.

I get that everyone's responses are thinking some big public thing. I'm thinking small toy implementation for my home network. The toy experiment is how to do what's trivial in Node with Python. Mainly because I like working with python. I think the answer might be: Python is the wrong tool for the job.

Having multiple nodejs processes is the same thing as having multiple python processes w/ regards to sharing state.

What you're referring to works equally well in the single-process case for both.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#54

Earlier quoted context omitted.

problem with Sanic that it does not implement streaming properly. so it is very easy to kill any Sanic process, 10-20 seconds. again any.

Would you mind elaborating?

check Sanic code, it loads whole incoming payload into memory before processing it. event for 404, so I can write very simple script that would consume all memory. and you can not really protect sanic service with proxy (nginx)

Re: Write Fast Apps Using Async Python 3.6 and Redis

#55

Earlier quoted context omitted.

You normally use something like redis to store the state. If you were going to share state in memory between threads, how would you handle the case where the second request goes to a different server or that the process has restarted? You'd need redis anyway, so you might as well just use it in all cases.

I get that everyone's responses are thinking some big public thing. I'm thinking small toy implementation for my home network. The toy experiment is how to do what's trivial in Node with Python. Mainly because I like working with python. I think the answer might be: Python is the wrong tool for the job.

[deleted]

Re: Write Fast Apps Using Async Python 3.6 and Redis

#56
post #48
post #42

Earlier quoted context omitted.

well it can also make things faster. well in your example it won't. but consider you need to load 4 requests and do operations on each of them. if you schedule them in an async fashion you can begin operating on the first one that's ready and not the first one you defined. and this is also often the case. a website does not just do one request to the database. mostly it runs multiple ones and often they don't interfe…

> if you schedule them in an async fashion you can begin operating on the first one that's ready and not the first one you defined. This type of operation is a given in any production quality webserver, whether it runs with multiple threads and blocking IO or using a non-blocking approach with greenlets. For a web application, this is an implementation detail that should not be explicit within the request handling co…

I think you're talking about different things; the idea is not that you can multiplex the requests coming in, but also the requests going out to the database and etc for each web request handling function.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#57
post #50

Earlier quoted context omitted.

I get that everyone's responses are thinking some big public thing. I'm thinking small toy implementation for my home network. The toy experiment is how to do what's trivial in Node with Python. Mainly because I like working with python. I think the answer might be: Python is the wrong tool for the job.

The problem is that accessing shared state concurrently in a multi-process context is a non-trivial problem, so specific software emerged that handles these problems for you. The simplest solution is to use a small DB system like sqlite. It is built into Python (import sqlite3) performs reasonably well and you do not have to run an additional service. Now if a small DB like sqlite already feels overblown to you (and…

I'd say the simplest solution is a global (or just shared) variable using a thread-safe container like queue.Queue. There's also multiprocessing.Queue, which supports sharing the queue across multiple workers.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#58

Earlier quoted context omitted.

You normally use something like redis to store the state. If you were going to share state in memory between threads, how would you handle the case where the second request goes to a different server or that the process has restarted? You'd need redis anyway, so you might as well just use it in all cases.

I get that everyone's responses are thinking some big public thing. I'm thinking small toy implementation for my home network. The toy experiment is how to do what's trivial in Node with Python. Mainly because I like working with python. I think the answer might be: Python is the wrong tool for the job.

This maybe seems complicated because Node has 1 obvious way to run (single threaded with asynchronous functions) but Python has a few ways (single threaded, multithreaded, ioloops kind of like Node, greenlets).

Python is excellent for toy implementations, and real ones too in many cases.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#59

Earlier quoted context omitted.

You normally use something like redis to store the state. If you were going to share state in memory between threads, how would you handle the case where the second request goes to a different server or that the process has restarted? You'd need redis anyway, so you might as well just use it in all cases.

I get that everyone's responses are thinking some big public thing. I'm thinking small toy implementation for my home network. The toy experiment is how to do what's trivial in Node with Python. Mainly because I like working with python. I think the answer might be: Python is the wrong tool for the job.

This maybe seems complicated because Node has 1 obvious way to run (single threaded with asynchronous functions) but Python has a few ways (single threaded, multithreaded, ioloops kind of like Node, greenlets).

Python is excellent for toy implementations, and real ones too in many cases.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#60
post #48

Earlier quoted context omitted.

> if you schedule them in an async fashion you can begin operating on the first one that's ready and not the first one you defined. This type of operation is a given in any production quality webserver, whether it runs with multiple threads and blocking IO or using a non-blocking approach with greenlets. For a web application, this is an implementation detail that should not be explicit within the request handling co…

I think you're talking about different things; the idea is not that you can multiplex the requests coming in, but also the requests going out to the database and etc for each web request handling function.

So on that topic, a request typically has a single transaction going out to the database so within the scope of the request, has to perform its steps in serial in any case. If it needs to make several requests to web services that aren't dependent on each other, that's an area where you can get into stacking them with some kind of concurrency construct (I'd pass it into a greenlet oriented worker pool). but this is already going to be a heavy web request with multiple web service calls.
Post reply on HN