Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the best tools for web scraping in 2022?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 160 posts

Re: Ask HN: What are the best tools for web scraping in 2022?

#91

Post by simonw »

shubhamjain wrote: post #88
simonw wrote: post #62

It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand. I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems. I wrote my own CLI scraping tool on top of Playwrig…

My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.

Yeah, my first step in trying to scrape an SPA is always to hit the network tab in the browser, filter by type=JSON and then sort by size. The largest responses are often the most useful, and can then be grabbed with curl.

Sometimes though that's not enough - particularly on older sites that might use weirder concepts like ASP.NET View state. For those I find having Playwright around is a big benefit.

Generally the things I have the most trouble with for non-browser-automation scraping are things with complex state stored in cookies and URL fragments (and maybe even localStorage these days).

Re: Ask HN: What are the best tools for web scraping in 2022?

#92

Post by shireboy »

I have had some luck running puppeteer in a nodejs app hosted at glitch.com. I spring for the (cheap) paid hosting and get several containers for dev/test/prod, web based ide. Obviously, this would only scale to a point. In my case I just need a single client automating interaction with a single site. If I really needed scale, I'd probably use one of the services listed elsewhere.

Of course, if you don't need a full javascript-enabled browser parse, consider alternatives first: simple HTTP requests, API, RSS, etc.

Re: Ask HN: What are the best tools for web scraping in 2022?

#93

Post by yobbo »

Many years ago I wrote a scraper-module for a scripting language that exposed a fake DOM to an embedded JS-engine, spidermonkey. The DOM was just an empty object graph, readable both from the scripting language and inside the JS context. The documents were parsed by libxml2 and the resulting DOMs were not identical to mozilla's, for example. But fast and efficient.

The purpose was to enable "live interactive" scraping of forms/js/ajax sites, with a web frontend controlling maybe 10 scrapers for each user. When that project fell through, I stopped maintaining it and the spidermonkey api has long since moved on.

It works for simple sites that don't require the DOM to actually do anything (for example triggering images to load with some magic url). But many simple DOM behaviours can be implemented.

Re: Ask HN: What are the best tools for web scraping in 2022?

#94

Post by Stefan_Smirnov »

intelVISA wrote: post #90
Stefan_Smirnov wrote: post #81

I’m biased since I’m an owner of a web scraping agency ( https://webscrapingsolutions.co.uk/ ). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy ( https://github.com/scrapy/scrapy ), Redis, PostgreSQL. for the following reasons: [1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of…

Is that even legal? I've built a few fast scrapers in C but I balk at the thought of selling them for some reason it feels a bit grey area to me.

It depends. We employ a lawyer to assess risks associated with each project.

In general - as long as you don't have to login, don't infringe on intellectual property rights and don't harm targeted servers - you should be ok.

Re: Ask HN: What are the best tools for web scraping in 2022?

#95

Post by latchkey »

simonw wrote: post #86
latchkey wrote: post #72

Earlier quoted context omitted.

Totally separate question, but I'm wondering why you put 'Mar' in your url instead of the month number?

It's a decision from 2003 I think. It's mainly because I'm from the UK, so I'm extremely sensitive to the risk of people confusing DD-MM-YYYY and MM-DD-YYYY - the least ambiguous format is to use DD-Mon-YYYY, so I picked that for my URLs. If I was designing my blog today I'd probably drop the day and month entirely, and go with /yyyy/unique-text-slug for the URLs.

Thanks simon! Makes perfect sense.

Re: Ask HN: What are the best tools for web scraping in 2022?

#96

Post by infinite8s »

simonw wrote: post #77
shubhamjain wrote: post #50

Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…

For things like regular expressions, it's useful to know that Python has a "-c" option which can be passed a multi-line string as part of a CLI pipeline. You can do something like this: curl 'https://news.ycombinator.com/' | python -c ' import sys, re, json html = sys.stdin.read() r = re.compile(" This outputs JSON which you can then pipe to other tools.

That's great! I didn't know -c supported multiline - I always just crammed it into one line with semicolons.

Re: Ask HN: What are the best tools for web scraping in 2022?

#97

Post by vitorbaptistaa »

namuorg wrote: post #51

I built a tool called Browserflow ( https://browserflow.app ) that lets you automate any task in the browser, including scraping websites. People love it for its ease-of-use because you can record actions via click-and-point rather than having to manually come up with CSS selectors. It intelligently handles lists, infinite scrolling, pagination, etc. and can run on both your desktop and in the cloud. Grateful for how…

It looks amazing! Great intro video.

It probably doesn't make sense for Browserflow as a business, but I'd love to find a tool like this that exported a Scrapy spider, similar to the now unmaintained Portia.

Re: Ask HN: What are the best tools for web scraping in 2022?

#98

Post by [deleted] »

sph wrote: post #7

As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control. Scrapers are very simple, effective and probably one of the least fun things to build.

[deleted]

Re: Ask HN: What are the best tools for web scraping in 2022?

#100

Post by simonw »

infinite8s wrote: post #96
simonw wrote: post #77

Earlier quoted context omitted.

For things like regular expressions, it's useful to know that Python has a "-c" option which can be passed a multi-line string as part of a CLI pipeline. You can do something like this: curl 'https://news.ycombinator.com/' | python -c ' import sys, re, json html = sys.stdin.read() r = re.compile(" This outputs JSON which you can then pipe to other tools.

That's great! I didn't know -c supported multiline - I always just crammed it into one line with semicolons.

Yeah I was the same - I only figured out the multi line trick a few days ago https://til.simonwillison.net/aws/boto-command-line
Post reply on HN