If you're familiar with Go, there's Colly too [1]. I liked its simplicity and approach and even wrote a little wrapper around it to run it via Docker and a config file: https://gotripod.com/insights/super-simple-site-crawling-and... [1] http://go-colly.org/
The go-colly framework was a bit too restrictive for my needs, but its very easy to build something on top of the standard lib's net/http, its cookiejar, and a third party library called goquery (afaik go-colly uses this too).
Fun Fact: We were scraping something from an apparantly zero rate limits azure blob container, and we had to enumerate around a million URLs daily (didn't know which URLs actually existed so we guessed an offset and enumerated from there, also we had to do it at a fixed time daily). We had proxys at our disposal but didn't need them cause the blob container did not rate-limit.
I wrote the scraper in Go, but a friend wrote it in Rust. Using Go was fast enough, satisfying all our requirements, but it turned out that the Rust one was 3-5 times faster. I tried to improve the Go scraper's speed by tweaking net/http transport's parameters, increasing workers, removing all NOFILE limits from SystemD and tried to profile and remove the low hanging speed issues. Nothing reduced the gap. Then I replaced the net/http client with valyala/fasthttp (another http implementation in Go) , which made it as fast as or slightly faster than the Rust one which was using the reqwest crate as http client.