Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee – Web scraping and browser automation library for Node.js

crawlee.dev

Post reply on HN
Print view View original

11–20 of 86 posts

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#11

Post by mnmkng »

franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

> While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff.

Love that comment :D

Yeah, the ability to switch between headless and http is very important to us in production. We often hack something up quickly with headless and then later optimize it to use HTTP when we find the time.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#12

Post by bilekas »

franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

> I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language

Sorry but this irked me. What exactly are you hangups with JS ? It's just a JiT dynamic typed language. By design.

It has its qwirks for sure, but again, its just a language. Truly awful it isn't.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#13

Post by jonatron »

OtmaneBenazzou wrote: post #3

Hi! It looks really REALLY cool! Is there any kind of detection/stealthiness benchmark compared to libraries such as puppeteer-stealth or fakebrowser? Honestly no matter how feature-complete and powerful a scraping tool is, the main "selling point" for me will always be stealthiness/human like behavior no matter how crappy the dev experience is.(and IMHO that's the same for most serious scrapers/bot makers) Will it a…

I'm not aware of a benchmark, but puppeteer-extra-plugin-stealth can be detected: https://datadome.co/bot-management-protection/detecting-head...

Crawlee does appear to do the basic checks though, like checking navigator.webdriver: https://github.com/apify/crawlee/blob/master/test/browser-po...

Last time I checked (over a year ago) I couldn't find any public code to make Chrome/Firefox properly undetectable.

That said, going to extreme lengths to be undetectable is rarely necessary, because some sites will serve up CAPTCHA's to real people on clean uncompromised residential connections anyway.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#15

Post by metalwarrior665 »

monkeydust wrote: post #8

Can I use this to log into LinkedIn, run a query on posts and then send me an email of the results? (In theory of course as I am sure this will violate some policy)

LinkedIn is one of the most protected websites out there so you always risk getting the account banned. But at small scale, it should be fine. Crawlee has support for Playwright + Firefox with statistically generated fingerprints (you can also pass your own fingerprint) which looks pretty human-like. Put some random sleeps in between actions so it looks like you are actually using your mouse.

To send emails, you can use any 3rd party tools, check out Apify as Crawlee is well integrated there and they have email sender easy to use.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#16

Post by alvis »

In a way, I hate you, but at the same time I love you. It's because I'm working on something similar to get data for my product. Seems like I'm going to use Ampify instead to save my life.

Just a feedback from the developer point of view tho. I think the documentation (both clawlee & Apify) need some work. I took me a while get the difference between clawlee & other headless crawler like playwight etc.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#17

Post by revskill »

franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

No post body was provided.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#18

Post by elevenoh »

Cool

One issue I have w/ webdriving headless browser in general is host RAM usage per browser/chromium/puppeteer instance (e.g. ~600-900mb) for a single browser/context/page.

Could crawlee make it easier to run more browser contexts with less ram usage?

e.g. concurrently running multiple of these (pages requiring js execution): https://crawlee.dev/docs/examples/forms

Post reply on HN