Live data from Hacker News

Show HN: Voyager – write a web crawler/scraper as a state machine in Rust

github.com

11–12 of 12 posts

Re: Show HN: Voyager – write a web crawler/scraper as a state machine in Rust

#11
post #9
post #8

Earlier quoted context omitted.

Yes, you can either limit how many request can be send concurrently or enforce a fixed or random delay in between consecutive requests. Basically this a just a futures_timer::Delay [0] that is reset after each request, which is non blocking. [0] https://docs.rs/futures-timer/3.0.2/futures_timer/

How do you set the concurrency limit? I'm down to use your framework (thanks!), just curious how you implement it. I couldn't get Stream::buffered to work correctly.

You can use `Futures::StreamExt::buffer_unordered` for this. [1] is an example where I used it for benchmark which creates a certain amount of QUIC connections at a time.

But you can also spawn all tasks upfront via `tokio::spawn`, and let them wait on a `tokio::sync::Semaphore` before making the request. The drawback of this is that you might allocate more memory for tasks upfront - but if you don't have an extremely high number it might not matter.

[1] https://github.com/quinn-rs/quinn/blob/de627437bc7d836564c36...

Re: Show HN: Voyager – write a web crawler/scraper as a state machine in Rust

#12
post #9

Earlier quoted context omitted.

How do you set the concurrency limit? I'm down to use your framework (thanks!), just curious how you implement it. I couldn't get Stream::buffered to work correctly.

You can use `Futures::StreamExt::buffer_unordered` for this. [1] is an example where I used it for benchmark which creates a certain amount of QUIC connections at a time. But you can also spawn all tasks upfront via `tokio::spawn`, and let them wait on a `tokio::sync::Semaphore` before making the request. The drawback of this is that you might allocate more memory for tasks upfront - but if you don't have an extremel…

This is a late comment, but this helped me out a lot. Thank you for the detailed explanation and code example, I appreciate it.
Post reply on HN