Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee – Web scraping and browser automation library for Node.js

crawlee.dev

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#71

Post by mnmkng »

leonhard wrote: post #70
mnmkng wrote: post #5

Earlier quoted context omitted.

Thanks, that's our experience exactly and that's why we built the library this way. It's not uncommon to switch from HTTP to headless back to HTTP in the lifecycle of a project as the website evolves or as you find better ways to scrape.

I‘m very new to web scraping, can you explain what the use cases are for each and how you can switch between them? As far as I understood you can use HTTP scraping for static websites and need some kind of browser/headless browser to scrape dynamically rendered websites. How would you do that with plain HTTP? By figuring out the ajax network requests and then sending those directly?

Exactly. The dynamic websites need to pull the data from somewhere as well. There's no magic behind it. Either all the data is in the initial payload in some form (not necessarily HTML), or it's downloaded later, again, over HTTP.

Headless browsers are useful when the servers are protected by anti-scraping software and you can't reverse engineer it, when the data you need is generated dynamically - not downloaded, but computed, or simply when you don't have the time to bother with understanding the website on a deeper level.

Usually it's a tradeoff between development costs and runtime costs. In our case, we always try plain HTTP first. If we can't find an obvious way to do it, we go with browsers and then get back to optimizing the scraper later, using plain HTTP or a combination of plain HTTP and browsers for some requests like logins, tokens or cookies.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#73

Post by mfrye0 »

This looks cool at first glance. I'll dig into it more.

One note that may be helpful, if all you care about is the HTML, it's better to take a "snapshot" of the page by streaming the response directly to blob storage like S3. That way if something fails and you need to retry, you can reference the saved raw data from storage vs making another request and potentially getting blocked. Node pipelines makes it really easy to chain this stuff together with other logic.

For reference, I run a company that does large scale scraping / data aggregation.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#74

Post by mdaniel »

dennisy wrote: post #72

Oh this looks lovely, congratulations! I would really like this but running in Python.

you want something better than Scapy? maybe I have Stockholm syndrome but I find it to be very well structured, testable, and has solved every problem I've had with running scrapers

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#75

Post by [deleted] »

bilekas wrote: post #12
franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

> I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language Sorry but this irked me. What exactly are you hangups with JS ? It's just a JiT dynamic typed language. By design. It has its qwirks for sure, but again, its just a language. Truly awful it isn't.

[deleted]

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#76

Post by mnmkng »

mfrye0 wrote: post #73

This looks cool at first glance. I'll dig into it more. One note that may be helpful, if all you care about is the HTML, it's better to take a "snapshot" of the page by streaming the response directly to blob storage like S3. That way if something fails and you need to retry, you can reference the saved raw data from storage vs making another request and potentially getting blocked. Node pipelines makes it really eas…

Yeah I agree, keeping the source HTML is great for debugging or retro-fixing issues. We also like to take screenshots on important errors, when running headless.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#77

Post by httbs »

MattyMc wrote: post #68
michaelmior wrote: post #66

This looks very cool, but am I the only one who has an aversion to any product/library calling itself the X instead of an X?

> Crawlee is a web scraping and browser automation library Above is the headline from the crawlee.dev website.

Yes but on Github, the place that people sensitive to the definite article will see the most, it says "The web scraping and browser automation library."

I opened a PR to change it: https://github.com/apify/crawlee/pull/1480

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#78

Post by LunaSea »

saiya-jin wrote: post #35
bilekas wrote: post #12

Earlier quoted context omitted.

> I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language Sorry but this irked me. What exactly are you hangups with JS ? It's just a JiT dynamic typed language. By design. It has its qwirks for sure, but again, its just a language. Truly awful it isn't.

node.js dependency hell that often makes java's transitive dependencies on bigger projects look like hello world? Exception handling? Performance? No real multithreading afaik? I have no skin in the game (anymore), but boy the feeling repeated gazillion times left and right in past decade about javascript crawling to be used in places it shouldn't be strongly resonated with me back then (not particularly for this pro…

> node.js dependency hell

You have one file which lists your dependencies and it comes as a standard with the framework compared to Java & co where you have lots of flavours. The amount of dependencies your dependencies have is up to your choice.

> Exception handling?

Synchronous code uses the standard try / catch method. Asynchronous code has been using async / await combined with try / catch. A strategy that Nose.js invented and that other languages like Java, Rust and Python copied.

> Performance?

V8 and Node.ja are pretty much the fastest dynamic language platform. Years ago, when companies started switching to Node.js, a lot of companies actually reduced the amount of servers they used when switching from Java to Node.js

> No real multithreading afaik?

I/O is multithreaded through libuv and C-Ares. The rest can simply run as a multi-process app. Worker pools have also recently been introduced. In any case that wouldn't be an issue for crawling which doesn't require multithreading.

JavaScript is the language used in the front-end so it seems like the most fitting language to use for crawling and scraping, especially since the introduction of headless browsers.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#79

Post by franga2000 »

nsonha wrote: post #58
franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

people trash JavaScript while using python? They should look at the mirror! Anyone can trash js except for all these C like languages with boringly similar designs, doubly so for python. Itself took over the world due to the fact that amateur (scientists and other professions as opposed to programmers) can easily play with it.

Python is anything but a C-like language, I really don't know what you're talking about.

Re: Show HN: Crawlee – Web scraping and browser automation library for Node.js

#80

Post by franga2000 »

bilekas wrote: post #12
franga2000 wrote: post #9

Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this…

> I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language Sorry but this irked me. What exactly are you hangups with JS ? It's just a JiT dynamic typed language. By design. It has its qwirks for sure, but again, its just a language. Truly awful it isn't.

Nothing against dynamically typed languages, Python is one as well. Maybe "awful" was a bit of a strong word, but to quickly summarise my issues with JS: - all the syntax and duck typing footguns (see all those "js is weird" memes) - the complexity of the build chain, especially if you want typing support - the unmanageable dependency mess that seems to be the norm in the ecosystem
Post reply on HN