This could be a nice benchmark for a http client. I wonder if anybody with experience with reqwest in Rust could chime in. Currently using aiohttp I can get around 1k requests/s from the Hacker News API, which means it will take around 9 hours to get all the items....
Ask HN: How fast can you hit all 34M items in Hacker News Firebase API?
1–6 of 6 posts
Re: Ask HN: How fast can you hit all 34M items in Hacker News Firebase API?
#2If it makes your experiment any easier, I have written a hacker news API wrapper in Python - https://github.com/avinassh/haxor
Re: Ask HN: How fast can you hit all 34M items in Hacker News Firebase API?
#3Isn't it rate limited?
Re: Ask HN: How fast can you hit all 34M items in Hacker News Firebase API?
#4Isn't it rate limited?
No. The Algolia API is, but the official API isn't.
Re: Ask HN: How fast can you hit all 34M items in Hacker News Firebase API?
#5That depends entirely on your network latency and connection speed, doesn't it? I think a better benchmark for a client would be generating pages/threads, because HN's data structures are so suboptimal for that even HN itself sometimes has trouble with it. So take a super long thread like https://news.ycombinator.com/item?id=12907201 and see how long it takes to render it fully.
Re: Ask HN: How fast can you hit all 34M items in Hacker News Firebase API?
#6I’ve done it maybe 2 years ago learning pythons asyncio. I remember the time is around 3.5 hours. I only perform the scrape maybe 5-6 times over a week to avoid overloading the server.