Ask HN: What are the best tools for web scraping in 2022?

#1

Post by pablohoffman » Wed, Aug 10, 2022, 9:46 AM UTC

Last time this question was asked on HN was in 2017 (https://news.ycombinator.com/item?id=15694118), a lot has changed in the last 5 years in the world of web scraping (legal landscape, antibot unblockers, data type specific APIs, etc), so I thought it may be a good idea to refresh this question and see what are the most popular tools used by the HN community these days.

Re: Ask HN: What are the best tools for web scraping in 2022?

#2

Post by SQL2219 » Wed, Aug 10, 2022, 10:46 AM UTC

powershell

Re: Ask HN: What are the best tools for web scraping in 2022?

#3

Post by snehesht » Wed, Aug 10, 2022, 10:58 AM UTC

In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices.

[1] https://github.com/microsoft/playwright

[2] https://github.com/puppeteer/puppeteer

Re: Ask HN: What are the best tools for web scraping in 2022?

#4

Post by dyml » Wed, Aug 10, 2022, 11:47 AM UTC

SQL2219 wrote: post #2

powershell

Can you expand on this? I’ve always found powershell hard to do right.

Re: Ask HN: What are the best tools for web scraping in 2022?

#5

Post by majkinetor » Wed, Aug 10, 2022, 11:56 AM UTC

dyml wrote: post #4

SQL2219 wrote: post #2

powershell

Can you expand on this? I’ve always found powershell hard to do right.

    $x = iwr https://google.com
    foreach ($link in $x.Links.href) { iwr $link -OutFile foo ...}

Re: Ask HN: What are the best tools for web scraping in 2022?

#6

Post by lwthiker » Wed, Aug 10, 2022, 12:28 PM UTC

curl-impersonate[1] is a curl fork that I maintain and which lets you fetch sites while impersonating a browser. Unfortunately, the practice of TLS and HTTP fingerprinting of web clients has become extremely common in the past ~1 year, which means a regular curl request will often return some JS challenge and not the real content. curl-impersonate helps with that.

[1] https://github.com/lwthiker/curl-impersonate

Re: Ask HN: What are the best tools for web scraping in 2022?

#7

Post by sph » Wed, Aug 10, 2022, 12:36 PM UTC

As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control.

Scrapers are very simple, effective and probably one of the least fun things to build.

Re: Ask HN: What are the best tools for web scraping in 2022?

#8

Post by larrydag » Wed, Aug 10, 2022, 12:38 PM UTC

The polite package using R is intended to be a friendly way of scraping content from the owner. "The three pillars of a polite session are seeking permission, taking slowly and never asking twice."

https://github.com/dmi3kno/polite

Re: Ask HN: What are the best tools for web scraping in 2022?

#9

Post by mynameismon » Wed, Aug 10, 2022, 12:41 PM UTC

snehesht wrote: post #3

In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer

Also is the ability to open the Networks panel, to snoop on requests and find the exact API call that you might need to perform your task, instead of having to pull in all of HTML/JS/CSS crap. As a lot of SPAs have essentially pushed everything behind JSON APIs, all information is usually one (authenticated) API call away.

Re: Ask HN: What are the best tools for web scraping in 2022?

#10

Post by epberry » Wed, Aug 10, 2022, 12:46 PM UTC

I've thought https://www.scrapingbee.com/ looked great, especially their auto rotation of IP addresses.