Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the best tools for web scraping in 2022?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 160 posts

Re: Ask HN: What are the best tools for web scraping in 2022?

#121

Post by pbowyer »

benbristow wrote: post #107
shubhamjain wrote: post #88

Earlier quoted context omitted.

My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.

My £0.02 It's usually easier to use an Android emulator like GenyMotion or a rooted Android phone and use HTTPToolkit and/or some certificate bypassing method using Frida or other and then explore APIs through their official apps. I've scraped loads of stuff through unofficial APIs before this way. Most developers don't ever expect people to do this so they're often a bit less secure too. Alternatively sometimes doin…

Have you had any luck with FB this way? There's local history groups I'd dearly like to back up for future generations - plus posts from 6 months+ ago are already hard to get to.

Re: Ask HN: What are the best tools for web scraping in 2022?

#122

Post by iforgotpassword »

giancarlostoro wrote: post #114
benbristow wrote: post #107

Earlier quoted context omitted.

My £0.02 It's usually easier to use an Android emulator like GenyMotion or a rooted Android phone and use HTTPToolkit and/or some certificate bypassing method using Frida or other and then explore APIs through their official apps. I've scraped loads of stuff through unofficial APIs before this way. Most developers don't ever expect people to do this so they're often a bit less secure too. Alternatively sometimes doin…

> Most developers don't ever expect people to do this so they're often a bit less secure too. Yikes.

I faintly remember a story from a couple years ago where some pizza ordering app simply changed some get parameter to paid=yes after the user completed the payment process. Guess what happened when the guy who poked around the app set that parameter to yes before doing the payment step....

Re: Ask HN: What are the best tools for web scraping in 2022?

#123

Post by sireat »

vikR0001 wrote: post #29
fersarr wrote: post #25

Selenium via Python is really useful too if you need to do a bit more (e.g clicks) than just fetching the html from the page.

Selenium can be very difficult to install when it comes to specific browser libraries.. Playwright, just as one example, is very easy to install.

Isn't it just pip install selenium then download the correct driver version for your browser (place it in a path or supply path when initializing client)?

I use Selenium every few months so I have to update the drivers but otherwise it is pretty painless.

Selenium is much slower than BS4 which is much preferred for static sites.

Re: Ask HN: What are the best tools for web scraping in 2022?

#124

Post by nlh »

I'm working on a personal project that involves A LOT of scraping, and through several iterations I've gotten some stuff that works quite well. Here's a quick summary of what I've explored (both paid and free):

* Apify (https://apify.com/) is a great, comprehensive system if you need to get fairly low-level. Everything is hosted there, they've got their own proxy service (or you can roll your own), and their open source framework (https://github.com/apify/crawlee) is excellent.

* I've also experimented with running both their SDK (crawlee) and Playwright directly on Google Cloud Run, and that also works well and is an order-of-magnitude less expensive than running directly on their platform.

* Bright Data nee Luminati is excellent for cheap data center proxies ($0.65/GB pay as you go), but prices get several orders of magnitude more expensive if you need anything more thorough than data center proxies.

* For some direct API crawls that I do, all of the scraping stuff is unnecessary and I just ping the APIs directly.

* If the site you're scraping is using any sort of anti-bot protection, I've found that ScrapingBee (https://www.scrapingbee.com/) is by far the easiest solution. I spent many many hours fighting anti-bot protection doing it myself with some combination of Bright Data, Apify and Playwright, and in the end I kinda stopped battling and just decided to let ScrapingBee deal with it for me. I may be lucky in that the sites I'm scraping don't really use JS heavily, so the plain vanilla, no-JS ScrapingBee service works almost all of the time for those. Otherwise it can get quite expensive if you need JS rendering, premium proxies, etc. But a big thumbs up to them for making it really easy.

Always looking for new techniques and tools, so I'll monitor this thread closely.

Re: Ask HN: What are the best tools for web scraping in 2022?

#125

Post by hartator »

We've built https://serpapi.com

We've invented the industry what you referring as "data type specific APIs"; APIs that abstract away all proxies issues, captcha solvings, various layouts support, even scrapping-related legal issues, and much more to a clean JSON response every single call. It was a lot of work but our success rate and response times are now rivaling non-scraping commercial APIs: https://serpapi.com/status

I think the next battle will be still legal despite all the wins in favor of scrapping public pages and common sense understanding this is the way to go. The EFF has been doing an amazing work in this world and we are proud to be a significant yearly contributor to the EFF.

Re: Ask HN: What are the best tools for web scraping in 2022?

#127

Post by contingencies »

Terretta wrote: post #110
simonw wrote: post #86

Earlier quoted context omitted.

It's a decision from 2003 I think. It's mainly because I'm from the UK, so I'm extremely sensitive to the risk of people confusing DD-MM-YYYY and MM-DD-YYYY - the least ambiguous format is to use DD-Mon-YYYY, so I picked that for my URLs. If I was designing my blog today I'd probably drop the day and month entirely, and go with /yyyy/unique-text-slug for the URLs.

ISO 8601 is least ambiguous, as there's no question of "endian-ness". https://en.wikipedia.org/wiki/ISO_8601 YYYY-MM-DD Yes, with year last, most of the world does it one way, but most of the audience are often from a part of the world that (a) thinks it's most of the world, and (b) does it the wrong way by shuffling endian-ness.

This. Two additional benefits of this approach are it sorts correctly and it's already standard in China, which is effectively a whole heap of the world's internet population.

Re: Ask HN: What are the best tools for web scraping in 2022?

#128

Post by matheusmoreira »

shubhamjain wrote: post #88
simonw wrote: post #62

It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand. I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems. I wrote my own CLI scraping tool on top of Playwrig…

My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.

This was my approach too and it's been working great. Nowadays data isn't rendered directly into HTML anymore, it gets downloaded from some JSON API endpoint. So I use network monitoring tools to see where it's coming from and then inferface with the endpoint directly. I essentially wrote custom clients for someone else's site. One of my scrapers is actually just curl piped into jq. Sometimes they change the API and I have to adapt but that's fine.

> I understand companies can put roadblocks to hinder this

Can you elaborate? I haven't run into any roadblocks yet but I'm not scraping big sites or sending a massive number of requests.

Post reply on HN