Show HN: Crawlee – Web scraping and browser automation library for Node.js

Hey HN,

This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production.

For details, see the short video [2] or read the announcement blog post [3].

Main features:

- Supports headless browsers with Playwright or Puppeteer

- Supports raw HTTP crawling with Cheerio or JSDOM

- Automated parallelization and scaling of crawlers for best performance

- Avoids blocking using smart sessions, proxies, and browser fingerprints

- Simple management and persistence of queues of URLs to crawl

- Written completely in TypeScript for type safety and code autocompletion

- Comprehensive documentation, code examples, and tutorials

- Actively maintained and developed by Apify—we use it ourselves!

- Lively community on Discord

To get started, visit https://crawlee.dev or run the following command: npx crawlee create my-crawler

If you have any questions or comments, our team will be happy to answer them here.

[1] https://crawlee.dev/

[2] https://www.youtube.com/watch?v=g1Ll9OlFwEQ

[3] https://blog.apify.com/announcing-crawlee-the-web-scraping-a...

Show HN: Crawlee – Web scraping and browser automation library for Node.js
crawlee.dev

Hi! It looks really REALLY cool!

Is there any kind of detection/stealthiness benchmark compared to libraries such as puppeteer-stealth or fakebrowser?

Honestly no matter how feature-complete and powerful a scraping tool is, the main "selling point" for me will always be stealthiness/human like behavior no matter how crappy the dev experience is.(and IMHO that's the same for most serious scrapers/bot makers)

Will it always be free or could it turn into a product/paid SaaS?(kind of like browserless) I'm kind of wondering if it's worth learning it if the next cool features are going to be for paying users only.

Is this something that you use internally or is it just a way to promote your paid products?

Thanks :)

Thanks, that's our experience exactly and that's why we built the library this way. It's not uncommon to switch from HTTP to headless back to HTTP in the lifecycle of a project as the website evolves or as you find better ways to scrape.

Hi there!

We dont have any benchmarks for Crawlee just yet, but we are working on those as we speak. We care deeply about bot detection, one of the features of Crawlee is generated fingerprints based on real browser data we gather - you can read more about it in the https://github.com/apify/fingerprint-suite repository, which is used under the hood in Crawlee. For scraping via HTTP requests (e.g. cheerio/jsdom), we develop library called got-scraping (https://github.com/apify/got-scraping), that tries to mimic real browsers while doing fast HTTP requests.

Crawlee is and always will be open source. It originated from the Apify SDK (http://sdk.apify.com), which is a library to support development of so called Actors on the Apify Platform (http://apify.com) - so you can see it as a way for us to improve the experience of our customers. But you can use it anywhere you want, we provide ready to use Dockerfiles for each template.

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework!

I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this to run NOW, you can rewrite it later" so much easier to handle.

While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff.