Ask HN: What are the best tools for web scraping in 2022?

We’ve built a freemium cloud RPA software focused on web scraping and monitoring, called Browse AI.

https://www.browse.ai

It lets you train a bot in 2 minutes. The bot will then open the site with rotating geolocated ip addresses, solve captchas, click on buttons and scroll and fill out forms, to get you the data you need.

It’s integrated with Google Sheets, Airtable, Zapier, and more.

We have a Google Sheets addon too which lets you run robots and get their results all in a spreadsheet.

We have close to 10,000 users with 1,000+ signing up every week these days. That made us raise a bit of funding from Zapier and others to be able to scale quicker and build the next version.

Not a full fledged scraper but IDS[1] has great heuristics to figure relevant content/information behind HTML code therefore lesser/no iterations needed in case frontend code changes.

Would be cool to reverse engineer it and probably plug it into some JS rendering testing solution (say Puppeteer, etc.)

[1] https://chrome.google.com/webstore/detail/instant-data-scrap...

My http://heliumhq.com is open source and gives you a very simple Python API:

  from helium import *
  start_chrome('github.com/login')
  write('user', into='Username')
  write('password', into='Password')
  click('Sign in')

To get started:

  pip install helium

Also, you need to download the latest ChromeDriver and put it in your PATH.

Have fun :-)

My £0.02

It's usually easier to use an Android emulator like GenyMotion or a rooted Android phone and use HTTPToolkit and/or some certificate bypassing method using Frida or other and then explore APIs through their official apps.

I've scraped loads of stuff through unofficial APIs before this way. Most developers don't ever expect people to do this so they're often a bit less secure too.

Alternatively sometimes doing a Global GitHub / Sourcegraph search you might find someone else who's done the hard work to reverse engineer an API and open-sourced it.

I am probably in the minority, but I try to outsource scraping whenever possible. It's too much grunt work: you have to constantly baby sit the crawlers that break because websites keep changing.

Personally, I use Indexed (https://www.indexedinc.com) because they are technical and reliable, although there are many other providers out there..