Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

131–140 of 145 posts

Re: The State of Web Scraping 2022

#131

Post by antonf »

ForHackernews wrote: post #119
digitcatphd wrote: post #72

Earlier quoted context omitted.

Good take, IMO ethically speaking we should not penalize scrapers themselves but do so based on their use. Scraping Facebook to make a clone of profiles shouldn’t be held to the same scrutiny of scraping Facebook to do an internal analysis of user demographics for research purposes.

Why should either be discouraged?

With cloned profiles (or any data obtained and shared without your consent) it will be harder for you to exercise your right to be forgotten, for example.

Re: The State of Web Scraping 2022

#132

Post by gmanis »

What does HN think of web scraping for the purpose of price comparison?

I’m asking this because I run a small side project to show prices across retailers for a very small niche. The users are very very happy. Even the vendors started contacting to be listed on the comparison.

But I am unable to make a business out of it other than few affiliate commission.

Re: The State of Web Scraping 2022

#133

Post by valar_m »

joe_91 wrote: post #2

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

Do you have any recommendations for the "good proxies" you mentioned?

Re: The State of Web Scraping 2022

#134

Post by magixx »

gmanis wrote: post #132

What does HN think of web scraping for the purpose of price comparison? I’m asking this because I run a small side project to show prices across retailers for a very small niche. The users are very very happy. Even the vendors started contacting to be listed on the comparison. But I am unable to make a business out of it other than few affiliate commission.

I worked for a company that did exactly this many years ago. (They were even able to parter with some retailer). Their product worked well yet they still went out of business long ago. To be honest, I don't see much value in such a service, not that it doesn't exist, it's just hard to justify paying for this data.

Re: The State of Web Scraping 2022

#135

Post by edmundsauto »

fareesh wrote: post #75
edmundsauto wrote: post #73

Earlier quoted context omitted.

One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.

Ah interesting, I end up doing this manually, i.e. File.write followed by what I want to scrape

I believe scrapy has somewhat intelligent cache control options - maybe it could be recreated in a few dozens of lines of code, maybe a few hundred. But there are a huge number of these types of features - it’s basically a Swiss Army knife.

Examples include rotating proxies, rotating user agent headers. Hooks to add in middleware for processing pipelines. CLI switches to change your data output format. Nice debugging and logging.

Other large scale features include distributed crawlers. Scheduling. Monitoring UI so you can see progress via a web UI.

It’s what I reach for first, because you can be up and running with your first scraper in an hour. By hand, that’s maybe 10 minutes - but if you want to iterate, and your first scraper is a v1 rather than final effort… i think it’s definitely worth it.

Re: The State of Web Scraping 2022

#137

Post by 1vuio0pswjnm7 »

digitcatphd wrote: post #70
yashasolutions wrote: post #41

Earlier quoted context omitted.

Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a…

Google is a crawler not a scraper, these are two totally different things

A crawl requires "extraction" of data from a web page, which according to Wikipedia is part of the definition of so-called "web scraping". Even if a crawler is using a sitemap.xml file, it still has to "scrape" (retrieve and extract from) that file first. It seems crawling always requires scraping.

If all the pages to be retrieved are known a priori, before retrieval begins, then one would likely call that "scraping". Whereas if not all pages are known before retrieval begins, then one would likely call that "crawling".

Re: The State of Web Scraping 2022

#138

Post by stef25 »

tomschwiha wrote: post #105
stef25 wrote: post #83

Earlier quoted context omitted.

On an eCommerce site I'm responsible for I changed some links from a GET to a POST. "BingPreview" continued hitting those links with GET requests, polluting my logs with 100s of "method not allowed" entries. So I blocked that UA from those links, nothing changed. Banned the bot all together, still hitting my site. This went on for well over a year.

I believe BingPreview is acting like a regular user - so it is not behaving like a Robot but like a user.

What does that mean exactly? An actual user can't be involved because the links that trigger a GET simply aren't there anymore. Therefore I assume it's a bot hitting faulty links it finds in its cache.

Re: The State of Web Scraping 2022

#139

Post by RobSm »

KieranMac wrote: post #90
RobSm wrote: post #81

Earlier quoted context omitted.

How many contracts google breaches scraping billions of pages every month?

Given the nuances of browsewrap contract enforceability, perhaps not as many as you suggest. The tricky part with navigating this gray area is knowing the likely circumstances when a contract of adhesion may give rise to an actual legal claim. There are patterns.

So in the scale of google, 'not many' would be some few million per month? And all is good then, right? Even you use their scrapped data probably daily and are totally fine with that, right?

You think google bots read contracts before scraping website? really? :) If you had any experience in creating websites and launching them online, you would know how fast and often they arrive and how they do not care about your TOS. So the real 'violation' numbers might be very scary...for you.

https://ironcladapp.com/journal/contract-management/are-brow...

Re: The State of Web Scraping 2022

#140

Post by Ian_Kerins »

megous wrote: post #126
Ian_Kerins wrote: post #45

Earlier quoted context omitted.

Some web scraping can be unethical, say for example if you are scraping a site solely to mirror their content and add zero value to the original content owner. However, there are a lot of web scraping use cases which are beneficial to the site being scraped and actually add value. Two examples: - Google: Ahrefs & SEMRush scrape Google so they can provide SEO analytics to companies looking to grow their companies. Goo…

> Some web scraping can be unethical, say for example if you are scraping a site solely to mirror their content and add zero value to the original content owner. Archiving is unethical?

Good point, wouldn't say archiving is unethical at all...I was thinking more along the lines of someone scraping a entire segment of a websites data and reproducing it 1 for 1 on their own site with zero value add.

I think we can't make broad statements saying that web scraping is ethical or unethical, it isn't that black or white. It really depends on what is being scraped, how is it being used, and the intention of the scraper.

Post reply on HN