Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making 1M requests with Python-aiohttp

pawelmhm.github.io

Post reply on HN
Print view View original

1–10 of 84 posts

Re: Making 1M requests with Python-aiohttp

#3

Post by JustSomeNobody »

ben_jones wrote: post #2

Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?

I guess I don't know how JS was any more "designed for" async than python was.

Re: Making 1M requests with Python-aiohttp

#4

Post by darpa_escapee »

ben_jones wrote: post #2

Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?

This is a genuine question: in what ways is Python's async implementation lacking? Could it have been baked in a better way?

In what ways do languages that were supposedly designed for async programming different than Python?

Python is definitely lacking an elegant interface for async programming.

Re: Making 1M requests with Python-aiohttp

#5

Post by afarrell »

ben_jones wrote: post #2

Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?

I found it easier than doing it in javascript because I could insert `import pdb;pdb.set_trace()` into the code and get an interactive debugger.

Supposedly you can do this in javascript by running node with a particular flag, then connecting to a port on localhost, and opening the chrome debugger. However, the multiple times I've tried throughout 2014-2016 has show that to be incredibly finnicky. It is especially frustrating when trying to insert a debugger into an automated test.

Re: Making 1M requests with Python-aiohttp

#6

Post by henryw »

Looks pretty interesting to do async on python. I once did something similar in node (async by default) with a few lines of code. I think I scraped 12 or 20 million real URLs in 8 hours on a $5 cloud VM. It was limited by network bandwidth.

Re: Making 1M requests with Python-aiohttp

#7

Post by bdarnell »

darpa_escapee wrote: post #4
ben_jones wrote: post #2

Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?

This is a genuine question: in what ways is Python's async implementation lacking? Could it have been baked in a better way? In what ways do languages that were supposedly designed for async programming different than Python? Python is definitely lacking an elegant interface for async programming.

I think that Python 3.5 now has a very elegant interface for async programming. I prefer Tornado to the standard library's asyncio, but the new keywords are nice for both packages (disclaimer: I'm the maintainer of Tornado).

The downsides have nothing to do with the design of the language. The problem is that introducing a new concurrency model late in a language's life splits the ecosystem. Most existing packages are synchronous, so if you want to build asynchronous systems you must avoid packages like requests, django, or sqlalchemy and find (or develop) asynchronous equivalents for the functionality you need.

Javascript has an advantage here not because the design of the language is especially well-suited for asynchronous programming, but because it never went through a synchronous/multi-threading phase. Every javascript package is designed for asynchronous use.

Re: Making 1M requests with Python-aiohttp

#8

Post by henryw »

JustSomeNobody wrote: post #3
ben_jones wrote: post #2

Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?

I guess I don't know how JS was any more "designed for" async than python was.

From https://developer.mozilla.org/en-US/docs/Web/JavaScript/Even...:

JavaScript has a concurrency model based on an "event loop". This model is quite different than the model in other languages like C or Java.

...

A very interesting property of the event loop model is that JavaScript, unlike a lot of other languages, never blocks. Handling I/O is typically performed via events and callbacks, so when the application is waiting for an IndexedDB query to return or an XHR request to return, it can still process other things like user input.

Re: Making 1M requests with Python-aiohttp

#10

Post by aaossa »

imaginenore wrote: post #9

1,000,000 requests in 52 minutes is just 320 req/sec. Am I missing something? What's so amazing about this? I just deployed some production feed that serves at 1955 requests/second on a cheap VPS in freaking PHP, one of the slowest languages out there.

Why you say is not amazing? Honestly curious here :)
Post reply on HN