Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the best tools for web scraping in 2022?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 160 posts

Re: Ask HN: What are the best tools for web scraping in 2022?

#41

Post by breno »

estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface.

It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others.

estela has been recently published as OSS under the MIT license:

https://github.com/bitmakerla/estela

More details about it can be found in the release blog post and the official documentation:

https://bitmaker.la/blog/2022/06/24/estela-oss-release.html

https://estela.bitmaker.la/docs/

estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap.

All kinds of feedback and contributions are welcome!

Disclaimer: I'm part of the development team behind estela :-)

Re: Ask HN: What are the best tools for web scraping in 2022?

#42

Post by snehesht »

snehesht wrote: post #3

In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer

If you're interested in running the puppeteer in containers, take a look at chrome-aws-lambda[1] and browserless docker container[2]

Not affiliated with browserless, but they do have a free/paid cloud service. https://www.browserless.io

[1] https://github.com/alixaxel/chrome-aws-lambda

[2] https://github.com/browserless/chrome

Re: Ask HN: What are the best tools for web scraping in 2022?

#44

Post by snehesht »

lofatdairy wrote: post #35
snehesht wrote: post #3

In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer

Highly recommend playwright (if I'm not mistaken most of the big developers from puppeteer were hired by MS to work on playwright). I run into significantly less await/async problems with playwright than I did with puppeteer and the codegen tool is super helpful as a first pass option.

Playwright integrates with lot of different browsers compared to puppeteer which just uses chrome.

Re: Ask HN: What are the best tools for web scraping in 2022?

#45

Post by mouse-linguist »

sph wrote: post #7

As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control. Scrapers are very simple, effective and probably one of the least fun things to build.

Do you have recommendations for platforms that monitor scraping fleets?

I occasionally code scrapers for quick data aggregation, but have trouble running anything long-term because it can be a chore to monitor. I've been looking into various options for self-hosting some sort of monitor/dashboard that can send alerts but haven't found anything satisfying yet.

Re: Ask HN: What are the best tools for web scraping in 2022?

#48

Post by breno »

mouse-linguist wrote: post #45
sph wrote: post #7

As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control. Scrapers are very simple, effective and probably one of the least fun things to build.

Do you have recommendations for platforms that monitor scraping fleets? I occasionally code scrapers for quick data aggregation, but have trouble running anything long-term because it can be a chore to monitor. I've been looking into various options for self-hosting some sort of monitor/dashboard that can send alerts but haven't found anything satisfying yet.

We released estela for this and other purposes, check it out, maybe it will suit your needs:

https://github.com/bitmakerla/estela

Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. It would be great to have feedback to consider it when prioritizing some over others :-)

Re: Ask HN: What are the best tools for web scraping in 2022?

#49

Post by benibela »

I wrote my own webscraper: https://videlibri.de/xidel.html

The main purpose was to submit HTML forms. You just say in which input fields something should be written and then it does the other things (i.e. download the page, find all other fields and their default values, build a HTTP request from all of them and send that ).

The last 5 years, I spent updating the XPath implementation to XPath/XQuery 3.1. The W3C has put a lot new stuff in the new XPath versions like JSON support or higher order functions, for some reason they decided to turn XPath into a Turing-complete functional programming language.

Re: Ask HN: What are the best tools for web scraping in 2022?

#50

Post by shubhamjain »

Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes.

The only thing I wish was present was better support for RegExes. Bash and most unix tools don't support PCRE which can severely limiting. Plus, sometimes you want to process text as a whole vs line-by-line.

I would also recommend Python's sh[4] module if Shell scripting isn't your cup of tea. You get best of both worlds: faster dev work with Bash utils, and a saner syntax.

[1]: https://github.com/ericchiang/pup

[2]: https://csvkit.readthedocs.io/en/latest/

[3]: https://stedolan.github.io/jq/

[4]: https://pypi.org/project/sh/

Post reply on HN