Ask HN: What are the best tools for web scraping in 2022? • Hacker News Board

Re: Ask HN: What are the best tools for web scraping in 2022?

#11

Post by bunnyfoofoo » Wed, Aug 10, 2022, 12:50 PM UTC

lwthiker wrote: post #6

curl-impersonate[1] is a curl fork that I maintain and which lets you fetch sites while impersonating a browser. Unfortunately, the practice of TLS and HTTP fingerprinting of web clients has become extremely common in the past ~1 year, which means a regular curl request will often return some JS challenge and not the real content. curl-impersonate helps with that. [1] https://github.com/lwthiker/curl-impersonate

Oh wow, thank you! I had no idea something like this existed. I was manually compiling against BoringSSL.

Re: Ask HN: What are the best tools for web scraping in 2022?

#12

Post by arinze11 » Wed, Aug 10, 2022, 12:53 PM UTC

if you are looking for pre-made tools and dont want to write any code, check our https://webautomation.io

Re: Ask HN: What are the best tools for web scraping in 2022?

#13

Post by asdadsdad » Wed, Aug 10, 2022, 12:53 PM UTC

Can I piggyback on the question and ask what are people scraping these days?

Re: Ask HN: What are the best tools for web scraping in 2022?

#14

Post by shmoogy » Wed, Aug 10, 2022, 12:55 PM UTC

Has anybody created anything similar to Portia for scraping? I'd love to self host or pay a nominal fee to allow my team to create / adjust scrapers via a UI

Re: Ask HN: What are the best tools for web scraping in 2022?

#15

Post by shmoogy » Wed, Aug 10, 2022, 12:56 PM UTC

asdadsdad wrote: post #13

Can I piggyback on the question and ask what are people scraping these days?

Competitor pricing, stock, leadtimes, promotions.

Re: Ask HN: What are the best tools for web scraping in 2022?

#16

Post by XzAeRosho » Wed, Aug 10, 2022, 12:57 PM UTC

epberry wrote: post #10

I've thought https://www.scrapingbee.com/ looked great, especially their auto rotation of IP addresses.

ScrappingBee is a fantastic paid solution that makes most of the pain of modern web scraping go away.

The prices are not very friendly though...

Re: Ask HN: What are the best tools for web scraping in 2022?

#17

Post by XzAeRosho » Wed, Aug 10, 2022, 12:58 PM UTC

shmoogy wrote: post #15

asdadsdad wrote: post #13

Can I piggyback on the question and ask what are people scraping these days?

Competitor pricing, stock, leadtimes, promotions.

Also product information updates/mistakes. Some companies like to make sure that the data that is being shown out there is accurate.

Re: Ask HN: What are the best tools for web scraping in 2022?

#18

Post by colinramsay » Wed, Aug 10, 2022, 1:00 PM UTC

For a particular type of scraping, we wrote SSScraper on top of Colly and it works really well:

https://github.com/gotripod/ssscraper/

Re: Ask HN: What are the best tools for web scraping in 2022?

#19

Post by smashah » Wed, Aug 10, 2022, 1:01 PM UTC

I've been using puppeteer as it's got a very established ecosystem. There are also puppeteer plugins that make it very powerful against captchas/detection/etc.

The worst thing about Puppeteer is chrome and it's bad memory management so I'm going to give playwright a spin soon.

Re: Ask HN: What are the best tools for web scraping in 2022?

#20

Post by altilunium » Wed, Aug 10, 2022, 1:01 PM UTC

Beautiful Soup gets the job done. I made several app by using it.

[1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity)

[2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia)

[3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers)

[4] https://github.com/altilunium/wi-page (Scrap wikipedia to get most active contributors that contribute to a certain article)

[5] https://github.com/altilunium/arachnid (Web scraper, optimized for wordpress and blogger)