Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the best tools for web scraping in 2022?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 160 posts

Re: Ask HN: What are the best tools for web scraping in 2022?

#81

Post by Stefan_Smirnov »

I’m biased since I’m an owner of a web scraping agency (https://webscrapingsolutions.co.uk/). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy (https://github.com/scrapy/scrapy), Redis, PostgreSQL. for the following reasons:

[1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of training. There are a lot of guides for beginners.

[2] Lots of features are already implemented and open-source, you won’t have to waste time & money on them.

[3] There is a strong community that can help with most of the questions (I don't think any other alternative has that).

[4] Scrapy developers are cheap. You will only need junior+ to middle level software engineers to pull out most of the projects. It’s not rocket since.

[5] Recruiting is easier: - there are hundreds of freelancers with relevant expertise - if you search on LinkedIn - there are hundreds of software developers that have worked with Scrapy in the past, and you don’t need that many - you can grow expertise in your own team quickly - developers are easily replaceable, even on larger projects - you can use the same developers on backend tasks.

[6] You don’t need a DevOps expertise in your web scraping team because Scrapy Cloud (https://www.zyte.com/scrapy-cloud/) is good and cheap enough for 99% of the projects.

[7] If you decide to have your own infrastructure, you can use https://github.com/scrapy/scrapyd.

[8] The entire ecosystem is well-well-maintained and steadily growing. You can integrate a lot of 3-rd party services into your project within hours: proxies, captcha solving, headless browsers, HTML parsing APIs.

[9] It’s easy to integrate your own AI/ML models into the scraping workflow.

[10]. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis.

[11] Commercial support is available. There are several companies that can develop you an entire project or take over an existing one - if you don’t have the time/don’t want to do it on your own.

We have built dozens of projects in multiple industries:

- news monitoring

- job aggregators

- real estate aggregators

- ecommerce (anything from 1 website, to monitoring prices on 100k+ domains)

- lead generation

- search engines in a specific niche (SEO, pdf files, ecommerce, chemical retail)

- macroeconomic research & indicators

- social media, NFT marketplaces, etc

So, most of the projects can be finished using these tools.

Re: Ask HN: What are the best tools for web scraping in 2022?

#83

Post by shubhamjain »

plainnoodles wrote: post #71
shubhamjain wrote: post #50

Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…

My main qualms with bash as a scripting language are that its syntax is not only kind of bonkers (no judgement, I know it's an old tool) but also just crazily unsafe. I link to a few high-profile things whenever people ask me why my mantra is "the time to switch your script from bash to python is when you want to delete things". >rm -rf /usr /lib/nvidia-current/xorg/xorg https://github.com/MrMEEE/bumblebee-Old-and-ab…

There are couple of flags you can use to mitigate the safety risks. `set -u`, for instance, will thrown an error if an unbound variable is used. I always start my scripts with

> set -euo pipefail

Here's a detail explaination of all the switches: https://gist.github.com/mohanpedala/1e2ff5661761d3abd0385e82....

I do agree though, it's not the best tool. But combining CLI utilities tends to be fast.

Re: Ask HN: What are the best tools for web scraping in 2022?

#84

Post by shubhamjain »

simonw wrote: post #77
shubhamjain wrote: post #50

Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…

For things like regular expressions, it's useful to know that Python has a "-c" option which can be passed a multi-line string as part of a CLI pipeline. You can do something like this: curl 'https://news.ycombinator.com/' | python -c ' import sys, re, json html = sys.stdin.read() r = re.compile(" This outputs JSON which you can then pipe to other tools.

This is great. perl also has one-liners [1] one can use, but I gave up dealing with perl's obscure syntax. This is much better.

[1]: http://novosial.org/perl/one-liner/

Re: Ask HN: What are the best tools for web scraping in 2022?

#85

Post by [deleted] »

shubhamjain wrote: post #50

Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…

[deleted]

Re: Ask HN: What are the best tools for web scraping in 2022?

#86

Post by simonw »

latchkey wrote: post #72
simonw wrote: post #62

It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand. I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems. I wrote my own CLI scraping tool on top of Playwrig…

Totally separate question, but I'm wondering why you put 'Mar' in your url instead of the month number?

It's a decision from 2003 I think. It's mainly because I'm from the UK, so I'm extremely sensitive to the risk of people confusing DD-MM-YYYY and MM-DD-YYYY - the least ambiguous format is to use DD-Mon-YYYY, so I picked that for my URLs.

If I was designing my blog today I'd probably drop the day and month entirely, and go with /yyyy/unique-text-slug for the URLs.

Re: Ask HN: What are the best tools for web scraping in 2022?

#87

Post by dbecks »

namuorg wrote: post #51

I built a tool called Browserflow ( https://browserflow.app ) that lets you automate any task in the browser, including scraping websites. People love it for its ease-of-use because you can record actions via click-and-point rather than having to manually come up with CSS selectors. It intelligently handles lists, infinite scrolling, pagination, etc. and can run on both your desktop and in the cloud. Grateful for how…

Beautifully done! Really amazing tool.

Re: Ask HN: What are the best tools for web scraping in 2022?

#88

Post by shubhamjain »

simonw wrote: post #62

It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand. I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems. I wrote my own CLI scraping tool on top of Playwrig…

My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.

Re: Ask HN: What are the best tools for web scraping in 2022?

#89

Post by randomcarbloke »

simonw wrote: post #86
latchkey wrote: post #72

Earlier quoted context omitted.

Totally separate question, but I'm wondering why you put 'Mar' in your url instead of the month number?

It's a decision from 2003 I think. It's mainly because I'm from the UK, so I'm extremely sensitive to the risk of people confusing DD-MM-YYYY and MM-DD-YYYY - the least ambiguous format is to use DD-Mon-YYYY, so I picked that for my URLs. If I was designing my blog today I'd probably drop the day and month entirely, and go with /yyyy/unique-text-slug for the URLs.

least ambiguous is ISO 86 since it is standardised, but I agree that alpha month is unambiguous.

Re: Ask HN: What are the best tools for web scraping in 2022?

#90

Post by intelVISA »

Stefan_Smirnov wrote: post #81

I’m biased since I’m an owner of a web scraping agency ( https://webscrapingsolutions.co.uk/ ). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy ( https://github.com/scrapy/scrapy ), Redis, PostgreSQL. for the following reasons: [1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of…

Is that even legal? I've built a few fast scrapers in C but I balk at the thought of selling them for some reason it feels a bit grey area to me.
Post reply on HN