Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee – Web scraping and browser automation library for Node.js

crawlee.dev

Post reply on HN
Print view View original

21–30 of 86 posts

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#24

Post by metalwarrior665 »

alvis wrote: post #16

In a way, I hate you, but at the same time I love you. It's because I'm working on something similar to get data for my product. Seems like I'm going to use Ampify instead to save my life. Just a feedback from the developer point of view tho. I think the documentation (both clawlee & Apify) need some work. I took me a while get the difference between clawlee & other headless crawler like playwight etc.

Yeah, I agree, the functionality of the library is very rich so the focus now for new docs is to continue making it more streamlined and easier to navigate.

Crawlee is basically a big wrapper around open source tools like Puppeteer, Playwright, Cheerio (I would not call these crawlers though as they don't have any logic for enqueueing requests etc.)

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#26

Post by tpaksoy »

daolf wrote: post #21

Good job to the APIfy team for this, this look very interesting!

If B4nan is around here from Apify, amazing work. On Crawlee, and on MikroORM. I especially use MikroORM extensively in production. One of the best, if not the best, ORM for NodeJS.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#27

Post by B4nan »

tpaksoy wrote: post #26
daolf wrote: post #21

Good job to the APIfy team for this, this look very interesting!

If B4nan is around here from Apify, amazing work. On Crawlee, and on MikroORM. I especially use MikroORM extensively in production. One of the best, if not the best, ORM for NodeJS.

Thanks!

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#28

Post by mvolfik »

OlikEten wrote: post #14

Nice! Finally a web scraping library for the programming language most websites use. It was about time lol

Thank you! And exactly as you say, using the same language as the websites gives us some advantages - we have HTTP-only (no client-side JS) crawlers based on the Cheerio library, which mimics jQuery API, and if you later find out that you need to use a full headless browser with Puppeteer, you can just call the utility function injectJquery, and there's very little you have to modify to keep your script working

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#29

Post by corford »

franga2000 wrote: post #6
OtmaneBenazzou wrote: post #3

Hi! It looks really REALLY cool! Is there any kind of detection/stealthiness benchmark compared to libraries such as puppeteer-stealth or fakebrowser? Honestly no matter how feature-complete and powerful a scraping tool is, the main "selling point" for me will always be stealthiness/human like behavior no matter how crappy the dev experience is.(and IMHO that's the same for most serious scrapers/bot makers) Will it a…

> for me will always be stealthiness/human like behavior no matter how crappy the dev experience is Can't say I agree. The biggest value for me is being able to respond to site changes quickly. Having a key bot offline for an extended period of time can be costly, so being able to update, test and deploy it quickly is a big selling point. The vast majority of sites, including major companies, have very rudimentary bo…

We do a lot of web scraping (hundreds of millions of requests, multiple terabytes of data per month) and have been using Crawlee - previously known as Apify SDK - since its v0.20 days. We adopted it for exactly this reason. It's extremely versatile and very pleasant to build on. The combination of Node, JS and Crawlee's modular SDK offers a sweet spot for scraping that imho is light years ahead of anything else.

Helps too that the apify devs themselves are nice and super responsive (we've had quite a few PRs merged over the last couple of years). The SDK code (and supporting libs like browser-tool, got-scraping) is clean and very easy to read/follow/extend (happy to hear too that the license is going to remain unchanged).

Post reply on HN