Ask HN: What are the best tools for web scraping in 2022?

Post by jamescampbell » Thu, Aug 11, 2022, 9:41 PM UTC

shubhamjain wrote: post #132

matheusmoreira wrote: post #128

Earlier quoted context omitted.

This was my approach too and it's been working great. Nowadays data isn't rendered directly into HTML anymore, it gets downloaded from some JSON API endpoint. So I use network monitoring tools to see where it's coming from and then inferface with the endpoint directly. I essentially wrote custom clients for someone else's site. One of my scrapers is actually just curl piped into jq. Sometimes they change the API and…

> Can you elaborate? I haven't run into any roadblocks yet but I'm not scraping big sites or sending a massive number of requests. Cloudflare Bot Protection[1] is a popular one. The website is guarded by a layer of code that needs to be executed before continuing. Normal browsers will follow through. It can be hard to bypass. [1]: https://www.cloudflare.com/pg-lp/bot-mitigation-fight-mode/

Post by jamescampbell » Thu, Aug 11, 2022, 9:47 PM UTC

sph wrote: post #7

As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control. Scrapers are very simple, effective and probably one of the least fun things to build.

Post by benbristow » Fri, Aug 12, 2022, 3:37 PM UTC

pbowyer wrote: post #121

benbristow wrote: post #107

Earlier quoted context omitted.

My £0.02 It's usually easier to use an Android emulator like GenyMotion or a rooted Android phone and use HTTPToolkit and/or some certificate bypassing method using Frida or other and then explore APIs through their official apps. I've scraped loads of stuff through unofficial APIs before this way. Most developers don't ever expect people to do this so they're often a bit less secure too. Alternatively sometimes doin…

Have you had any luck with FB this way? There's local history groups I'd dearly like to back up for future generations - plus posts from 6 months+ ago are already hard to get to.

Post by mateuszbuda » Sat, Aug 13, 2022, 10:41 PM UTC

XzAeRosho wrote: post #16

epberry wrote: post #10

I've thought https://www.scrapingbee.com/ looked great, especially their auto rotation of IP addresses.

ScrappingBee is a fantastic paid solution that makes most of the pain of modern web scraping go away. The prices are not very friendly though...

Post by david_databar » Wed, Aug 24, 2022, 2:35 PM UTC

shireboy wrote: post #92

I have had some luck running puppeteer in a nodejs app hosted at glitch.com. I spring for the (cheap) paid hosting and get several containers for dev/test/prod, web based ide. Obviously, this would only scale to a point. In my case I just need a single client automating interaction with a single site. If I really needed scale, I'd probably use one of the services listed elsewhere. Of course, if you don't need a full…