41–50 of 86 posts
It would be very useful if this or some other library came with Captcha solvers or a way to add Captcha solvers to the scrapers. Even regular users get Captchas sometimes.
It looks like you can specify HTTP headers, so you should be able to use a captcha solving service with Crawlee.
Sweet that you went down the free route and made it an npm package, following the good way, by providing an optional upgrade to SaaS. Cool stuff. I could have used this dearly last time I scraped. Like others I used mixed methods (headless browser for renders and direct calls) and wrote a lot of error handling boilerplate.
I've been using the unmaintained node-osmosis lib for years, maybe it'll motivate me to finally move from it.
This seems great. I've been using the unmaintained node-osmosis lib for years, maybe it'll motivate me to finally move from it.
Looks great! Just wondering why it has a few scrapers built in - like puppeteer and cheerio. Is it because you might want headless only sometimes?
This looks really great. However, I can't find examples of how to handle scraping behind a login or a paywall, without having to 'type' credentials every time.
Is there a similar guide for Crawlee?
This looks really great. However, I can't find examples of how to handle scraping behind a login or a paywall, without having to 'type' credentials every time.