Probably the best tool for scraping websites protected by services like cloudflare. https://github.com/ultrafunkamsterdam/undetected-chromedrive...
Wow, thanks!
Ask HN: What are the best tools for web scraping in 2022?
141–150 of 160 posts
Re: Ask HN: What are the best tools for web scraping in 2022?
#142Earlier quoted context omitted.
It's a decision from 2003 I think. It's mainly because I'm from the UK, so I'm extremely sensitive to the risk of people confusing DD-MM-YYYY and MM-DD-YYYY - the least ambiguous format is to use DD-Mon-YYYY, so I picked that for my URLs. If I was designing my blog today I'd probably drop the day and month entirely, and go with /yyyy/unique-text-slug for the URLs.
ISO 8601 is least ambiguous, as there's no question of "endian-ness". https://en.wikipedia.org/wiki/ISO_8601 YYYY-MM-DD Yes, with year last, most of the world does it one way, but most of the audience are often from a part of the world that (a) thinks it's most of the world, and (b) does it the wrong way by shuffling endian-ness.
The killer feature of ISO-8601 style dates is that you don't need to parse the dates to sort them. Lexicographical order is chronological order. That's a pretty huge deal.
Re: Ask HN: What are the best tools for web scraping in 2022?
#143Earlier quoted context omitted.
This was my approach too and it's been working great. Nowadays data isn't rendered directly into HTML anymore, it gets downloaded from some JSON API endpoint. So I use network monitoring tools to see where it's coming from and then inferface with the endpoint directly. I essentially wrote custom clients for someone else's site. One of my scrapers is actually just curl piped into jq. Sometimes they change the API and…
> Can you elaborate? I haven't run into any roadblocks yet but I'm not scraping big sites or sending a massive number of requests. Cloudflare Bot Protection[1] is a popular one. The website is guarded by a layer of code that needs to be executed before continuing. Normal browsers will follow through. It can be hard to bypass. [1]: https://www.cloudflare.com/pg-lp/bot-mitigation-fight-mode/
So that would break text browsers too, right? :(
And users with JS disabled for privacy reasons.
Re: Ask HN: What are the best tools for web scraping in 2022?
#144It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand. I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems. I wrote my own CLI scraping tool on top of Playwrig…
"It's increasingly difficult these days to write scrapers that don't at some point need to execute Javascript on a page - so you need to have a good browser automation tool on hand." But doesn't this assume which sites are being "scraped". How would anyone know what sites someone else needs to "scrape" unless people name the sites (and the specific pages at the sites as this is not "crawling"). For example, none of t…
I'll rephrase:
"It's increasingly difficult these days to regularly write scrapers for a large range of different websites without eventually hitting a situation where you need to execute JavaScript on a page"
Re: Ask HN: What are the best tools for web scraping in 2022?
#145* Shameless plug *: our super-easy feed builder at New Sloth (formerly Feedity) - https://newsloth.com combines a scraper and data transformer, which helps create custom RSS feeds for any public webpage. Our API can auto-magically detect relevant articles in most cases. The platform includes an integrated feed reader and clusterer/deduplicator, specially aimed for knowledge workers with hundreds and thousands of feeds to monitor daily.
Re: Ask HN: What are the best tools for web scraping in 2022?
#146As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control. Scrapers are very simple, effective and probably one of the least fun things to build.
Apologies for the shameless self-promotion here but it was this very problem that I built puppeteer-heap-snapshot. It decouples the HTML from the scraper and instead we inspect the booted app’s memory. Not near as performant but a lot more reliable. I wrote about it here: https://www.adriancooney.ie/blog/web-scraping-via-javascript...
This is amazing thanks for sharing!!
Re: Ask HN: What are the best tools for web scraping in 2022?
#147It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand. I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems. I wrote my own CLI scraping tool on top of Playwrig…
What happens in Playwright with sites using CAPTCHAs?
I had been occasionally scraping a site via curl, but then they started using Cloudflare's anti-bot stuff.
I switched to Selenium and that worked for a while--my Selenium script would navigate to the site, pause to let me manually deal with Cloudflare, and then automatically grab the data I wanted. But then that stopped working.
I found a Stack Overflow answer that gave some settings in Selenium to make it not tell the site's JavaScript that the browser was being automated and that briefly made things happy, but not too long afterwards that broke. There's a Selenium Chrome drive available that is meant for scraping which apparently tries to hide all evidence that the browser is being automated, but it didn't fool Cloudflare.
What I want is a browser-based automation tool that to the site is indistinguishable from a human browsing, except possibly by the timing of user actions. E.g., if the site can deduce it is being automated because the client responds faster than human reaction time, or with too little variation in response time, that's fine.
Re: Ask HN: What are the best tools for web scraping in 2022?
#148Earlier quoted context omitted.
> Most developers don't ever expect people to do this so they're often a bit less secure too. Yikes.
I faintly remember a story from a couple years ago where some pizza ordering app simply changed some get parameter to paid=yes after the user completed the payment process. Guess what happened when the guy who poked around the app set that parameter to yes before doing the payment step....
He went to jail?
Re: Ask HN: What are the best tools for web scraping in 2022?
#149Earlier quoted context omitted.
My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.
Totally agree! I've done this method a lot. Honestly scraping Google Reviews was the most difficult in terms of complexity. This was like 6 or 7 years ago. You would get back these huge nested arrays that mostly had 0s in them. Occasionally a value would be set and that's what I would go with. I'm assuming their internal tools were obfuscated and/or using protobuf. But it certainly took me back to the good ol' days h…
I was thinking you talking about Instagram :) i knows guy who has similar problems with them.
Re: Ask HN: What are the best tools for web scraping in 2022?
#150Perl + Mojo::DOM, or Perl + Playwright or Selenium when JavaScript is required