Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping in 2021

mihaisplace.blog

Post reply on HN
Print view View original

101–110 of 132 posts

Re: The State of Web Scraping in 2021

#101

Post by heavyset_go »

dreyfan wrote: post #48
bryanrasmussen wrote: post #15

Earlier quoted context omitted.

so will Google's freezing of the UA lead to less ability to web scrape for the non big company scrapers out there?

What? It’s just a text string in the header. How in the world would that possibly make it more difficult to scrape? All Chrome is doing is stop appending the current semver in the UA it sends.

The switch from UA to browser fingerprinting makes it harder to scrape without being stopped.

Yes, at any time the UA could be ignored and clients could be fingerprinted, but now the UA is being made next to useless, so fingerprinting will now become the default everywhere.

Re: The State of Web Scraping in 2021

#103

Post by chinchilla2020 »

tmpz22 wrote: post #45
dec0dedab0de wrote: post #42

Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…

To be fair selenium style scraping can take a lot of time to setup if you aren’t already familiar with the tooling, and the browser rendering apis are unintuitive and sometimes flat out broken.

I was surprised not to see selenium in this article. It is a common tool

Re: The State of Web Scraping in 2021

#104

Post by jamesfinlayson »

lloydatkinson wrote: post #77
novaleaf wrote: post #76

Self promotion: my SaaS is the lowest cost web scraping tool for high volume, and has been in business since 2016. https://PhantomJsCloud.com My SaaS requires some technical knowledge to use (call a web api) which I suppose is why it's not ever in these lists. Some of my customers are *very* large businesses. If you are looking at evading bot countermeasures, my product isn't probably the best for you. but for TCO no…

Isn't phantomjs deprecated and unmaintained?

For some time now - since 2016 I think (though someone briefly tried to revive it) - headless Chrome does it faster and better now.

Re: The State of Web Scraping in 2021

#106

Post by jamesfinlayson »

dec0dedab0de wrote: post #42

Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…

I remember a workmate having to deal with some difficult to scrape data at a previous job - the page randomly rendered with different mark-up (but the same appearance) to mitigate pulling out data using selectors. I think he got to the bottom of it eventually but it made testing his work a pain.

Re: The State of Web Scraping in 2021

#107

Post by omneity »

marban wrote: post #67

Cloudflare's protection is quite a b*tch to circumvent with any headless or python library.

Slight aside: The most recent Cloudflare HCaptchas ask you to classify AI generated images. They don’t even look like a proper bike/truck/whatever (I don’t have an example handy).

I categorically refuse to do when I’m browsing websites using it. I find this new captcha utterly unacceptable.

It’s no “protection” at this point anymore. Websites are using it as an excuse to become even more user hostile. I am worried for the future of the web.

Re: The State of Web Scraping in 2021

#108

Post by Nextgrid »

prox wrote: post #20
Ozzie_osman wrote: post #10

Earlier quoted context omitted.

Fundamentally, what most scrapers learn is that the more their scraper can behave like a human browsing the site, the less likely they are to get detected and blocked. This does put limits on how quickly they can crawl, of course, but scrapers find ways around it like changing ip and user agent (ip is probably the main one, bec you can then pretend that you are multiple humans browsing the site normally).

Yeah, there are services that give you a range of IPs for a certain time.

Even changing IPs won't always work against an adversary with a global view of the Internet such as CloudFlare.

CF has a view on a significant chunk of internet traffic across many sites and feeds that into some kind of heuristics/machine learning. Even if we assume that your behavior on the scraped website looks human-like, you may still get blocked or challenged because of your lack of traffic on other sites.

The IPs you'd get from a typical proxy service would only be used for bot activity and would've been classified as such a long time ago, and there's no "human activity" on it to compensate and muddy the waters so to speak.

The best solution is to use IPs with a chunk of legitimate residential traffic, and keep scraping sessions constrained to their IPs - don't rotate your requests among all these IPs, instead every IP should be its own instance of a human-like scraper, using its own user account, browser cookies, etc.

Re: The State of Web Scraping in 2021

#109

Post by 1vuio0pswjnm7 »

timwis wrote: post #96
eastendguy wrote: post #85

Earlier quoted context omitted.

> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.

Have you tried on a page protected by cloudflare captcha?

Its funny I never seem to hit these infamous Clouflare captchas. The only impediment I encounter with Cloudflare is they require plaintext SNI to read their blog, https://blog.cloudflare.com. Unlike almost all other Cloudflare, ESNI will not work.

Re: The State of Web Scraping in 2021

#110

Post by nlh »

Quessked73 wrote: post #88

Does anyone have a resource for getting into app-based scraping, if the API is obfuscated or rate limited?

ProxyMan on MacOS is quite awesome for this. It requires a bit of setup with certificates, etc., but once it's working you just fire up the target app on your phone and all the sweet sweet API requests appear on your big screen. I've scraped two apps very successfully this way.

It's also fascinating to see how developers-who-aren't-me setup their APIs when they assume that nobody's looking.

Post reply on HN