Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee – Web scraping and browser automation library for Node.js

crawlee.dev

Post reply on HN
Print view View original

31–40 of 86 posts

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#31

Post by mnmkng »

elevenoh wrote: post #18

Cool One issue I have w/ webdriving headless browser in general is host RAM usage per browser/chromium/puppeteer instance (e.g. ~600-900mb) for a single browser/context/page. Could crawlee make it easier to run more browser contexts with less ram usage? e.g. concurrently running multiple of these (pages requiring js execution): https://crawlee.dev/docs/examples/forms

In crawlee, you can use the useIncognitoPages option to create a separate context for each page https://crawlee.dev/api/browser-pool/class/LaunchContext#use... Not sure if it will be enough to offset your RAM requirements.

From our experience, RAM is not the limiting factor. It's the CPU. You need at least 1 CPU core for the modern browsers to work reliably at scale so if you're using a container that has 1GB ram and 0.25 core, it's just not worth it. If you have access to containers that have strong CPUs and not a lot of RAM, then it's a different story.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#32

Post by metalwarrior665 »

elevenoh wrote: post #18

Cool One issue I have w/ webdriving headless browser in general is host RAM usage per browser/chromium/puppeteer instance (e.g. ~600-900mb) for a single browser/context/page. Could crawlee make it easier to run more browser contexts with less ram usage? e.g. concurrently running multiple of these (pages requiring js execution): https://crawlee.dev/docs/examples/forms

That being said, for scraping purposes, you can almost always build the scraper with HTTP requests only. Sometimes it might be hard but theoretically it is always possible (it is what the browser itself does right).

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#33

Post by nodoodles »

Jan, thanks for the open approach to running the tech behind apify!

The libraries look useful - one question which wasn't obvious in the doc, how do you manage / suggest approaching rate limiting by domain? Ideally respecting crawl-delay in robots.txt, or just defaulting to some sane value.. most naive queue implementations make it challenging, and queue-per-domain feels annoying to manage.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#34

Post by quickthrower2 »

Sweet that you went down the free route and made it an npm package, following the good way, by providing an optional upgrade to SaaS. Cool stuff. I could have used this dearly last time I scraped. Like others I used mixed methods (headless browser for renders and direct calls) and wrote a lot of error handling boilerplate.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#35

Post by saiya-jin »

bilekas wrote: post #12
franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

> I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language Sorry but this irked me. What exactly are you hangups with JS ? It's just a JiT dynamic typed language. By design. It has its qwirks for sure, but again, its just a language. Truly awful it isn't.

node.js dependency hell that often makes java's transitive dependencies on bigger projects look like hello world? Exception handling? Performance? No real multithreading afaik?

I have no skin in the game (anymore), but boy the feeling repeated gazillion times left and right in past decade about javascript crawling to be used in places it shouldn't be strongly resonated with me back then (not particularly for this project, rant in general).

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#36

Post by mnmkng »

nodoodles wrote: post #33

Jan, thanks for the open approach to running the tech behind apify! The libraries look useful - one question which wasn't obvious in the doc, how do you manage / suggest approaching rate limiting by domain? Ideally respecting crawl-delay in robots.txt, or just defaulting to some sane value.. most naive queue implementations make it challenging, and queue-per-domain feels annoying to manage.

The ideal approach would depend on your architecture. It's really easy and cheap to create new queues on the Apify platform (we create ~500k every day) so we usually run a crawler per domain. It performs the best and it's the easiest to set up.

On Crawlee level, you can open new queues with one line of code and name them with the hostname, so the most straightforward solution would be to run multiple Crawler instances with multiple queues and then rate limit using the options explained here https://crawlee.dev/docs/guides/scaling-crawlers and push the new URLs to the respective queues using the URLs' hostname.

If you'd like to discuss this a bit more in depth, you can join our Discord or ask in GitHub discussions. Both are linked from Crawlee homepage.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#37

Post by elevenoh »

mnmkng wrote: post #31
elevenoh wrote: post #18

Cool One issue I have w/ webdriving headless browser in general is host RAM usage per browser/chromium/puppeteer instance (e.g. ~600-900mb) for a single browser/context/page. Could crawlee make it easier to run more browser contexts with less ram usage? e.g. concurrently running multiple of these (pages requiring js execution): https://crawlee.dev/docs/examples/forms

In crawlee, you can use the useIncognitoPages option to create a separate context for each page https://crawlee.dev/api/browser-pool/class/LaunchContext#use... Not sure if it will be enough to offset your RAM requirements. From our experience, RAM is not the limiting factor. It's the CPU. You need at least 1 CPU core for the modern browsers to work reliably at scale so if you're using a container that has 1GB ram and…

No post body was provided.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#39

Post by willmorrison »

bobajeff wrote: post #38

It would be very useful if this or some other library came with Captcha solvers or a way to add Captcha solvers to the scrapers. Even regular users get Captchas sometimes.

It looks like you can specify HTTP headers, so you should be able to use a captcha solving service with Crawlee.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#40

Post by mnmkng »

bobajeff wrote: post #38

It would be very useful if this or some other library came with Captcha solvers or a way to add Captcha solvers to the scrapers. Even regular users get Captchas sometimes.

You can use any captcha solving service with Crawlee, but we plan to add a plugin to make its use much easier. It's on our roadmap.
Post reply on HN