Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee – Web scraping and browser automation library for Node.js

crawlee.dev

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#61

Post by uwuemu »

boesboes wrote: post #59

I see you basically recommend bypassing rate limits by using proxies etc? Why not just respect rate limits if set properly? A little bit of consideration for what/whomever is on the other end ;)

Because everyone "being nice" is how Google keeps its monopoly on search. Googlebot can do anything and everything and no one complains. Or how about sites like Twitter and Instagram that live off selling their user's data having extreme limitations on their public apis and aggressively blocking alternative frontends like nitter or bibliogram because OF COURSE god forbid someone could want to look up something on their platform and not have an account.

The typical response to people raising these issues is "buuuut xy is a private platform that can do what it wants", yes, but why are you defending technocrats with bigger profits than many nation state's GDPs? (Reasonable) crawling should be allowed and promoted, in fact, it should be codified in law as a necessary element for the future of open and free internet. Anyone trying to prevent it, or even worse, make it illegal, is a bad actor.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#62

Post by wumpus »

nodoodles wrote: post #33

Jan, thanks for the open approach to running the tech behind apify! The libraries look useful - one question which wasn't obvious in the doc, how do you manage / suggest approaching rate limiting by domain? Ideally respecting crawl-delay in robots.txt, or just defaulting to some sane value.. most naive queue implementations make it challenging, and queue-per-domain feels annoying to manage.

Most mentions of crawl-delay in robots.txt set a limit so slow that the website can't be fully crawled before the heat death of the Universe. That's why Google and bing etc. ignore crawl-delay.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#68

Post by MattyMc »

michaelmior wrote: post #66

This looks very cool, but am I the only one who has an aversion to any product/library calling itself the X instead of an X?

> Crawlee is a web scraping and browser automation library

Above is the headline from the crawlee.dev website.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#69

Post by mnmkng »

cebert wrote: post #65
mnmkng wrote: post #64

Earlier quoted context omitted.

Yes

Awesome! Thanks.

If it doesn’t, please make an issue. We know it works from the community but we don’t have tests specifically for Lambda. But it should work, so we’ll help if it doesn’t.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#70

Post by leonhard »

mnmkng wrote: post #5
samwillis wrote: post #4

This looks really neat, I love the idea of a single api for both traditional and headless scraping. From my experience headless scraping is in the order of 10-100x slower and significantly more resource intensive, even if you carefully block requests for images/ads/etc. You should always start with traditional scraping, try as hard as you can to stick with it, and only move to headless if absolutely necessary. Someti…

Thanks, that's our experience exactly and that's why we built the library this way. It's not uncommon to switch from HTTP to headless back to HTTP in the lifecycle of a project as the website evolves or as you find better ways to scrape.

I‘m very new to web scraping, can you explain what the use cases are for each and how you can switch between them? As far as I understood you can use HTTP scraping for static websites and need some kind of browser/headless browser to scrape dynamically rendered websites. How would you do that with plain HTTP? By figuring out the ajax network requests and then sending those directly?
Post reply on HN