Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

31–40 of 145 posts

Re: The State of Web Scraping 2022

#31

Post by Ian_Kerins »

KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Interesting!...I'm not a lawyer, so the content for this piece was based on commentary in the below article. Was written by their lawyer, but would love to hear your counter point to it. Always good to get multiple viewpoints on something.

https://www.zyte.com/blog/van-buren-a-victory-for-web-scrape...

Re: The State of Web Scraping 2022

#32

Post by dewey »

ok_coo wrote: post #22

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/

I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.

Re: The State of Web Scraping 2022

#33

Post by mycall »

xrendan wrote: post #24
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

- Scraping faces to find missing persons: ethical

- Scraping photos to create deep learning VQGAN+CLIP art generator: ethical

.. we can go on and on, but we should all agree scraping is a useful tool that should never be outlawed.

Re: The State of Web Scraping 2022

#34

Post by charcircuit »

blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

Here are two use cases why I scrape YouTube.

- There is no external API for getting scheduled streams or when they have gone live AFAIK. This lets me be notified of new stuff to watch.

- The API for getting a channel's members is locked down. I applied for access to it 6 months ago and haven't heard anything about it from YouTube so I just scrape it to give members perks.

Re: The State of Web Scraping 2022

#35

Post by Mezzie »

lambic wrote: post #28
xrendan wrote: post #24

Earlier quoted context omitted.

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

The first one here is important. Despite the open data movement pressuring governments to provide their data in easily consumable forms, a lot of government organizations are still unable or unwilling to do so. Political advocacy orgs rely a lot on scraping to collect political representative data that isn't available through any other means.

Yes, and so do research orgs. My organization does a lot of scraping because we deal with local election data and that's. Uh. Let's just say that if all counties had websites that were like Web 1.0, that would be an improvement over the current situation.

Re: The State of Web Scraping 2022

#36

Post by zffr »

xrendan wrote: post #24
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

> - Scraping to just copy someone's data they worked hard to generate to go and resell: unethical

Wanted to include a slightly different application:

- Scraping multiple websites and organizing data in a new and useful way for customers: To me this would be ethical since it produces new value and does not just copy someone else's data as-is

Re: The State of Web Scraping 2022

#37

Post by joe_91 »

ok_coo wrote: post #22

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/

That looks like a great resource! How often is the data set "updated"?

I'd imagine most people's use cases need data which can change from day to day or week to week but I do think that this is fantastic if I was to have a project which was looking at data across a longer timeframe.

Re: The State of Web Scraping 2022

#38

Post by mycall »

emptysea wrote: post #17
joe_91 wrote: post #2

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

Is the “pretty optimized headless browser” an off the shelf thing, or something custom? Are you using playwright/puppeteer to drive it?

Headless Chrome [0] and alpine-Chrome [1] are pretty popular. Some variations also include V2Ray, Shadowsocks and other VPNs.

[0] https://hub.docker.com/r/justinribeiro/chrome-headless/

[1] https://github.com/Zenika/alpine-chrome

Re: The State of Web Scraping 2022

#39

Post by rozenmd »

emptysea wrote: post #17
joe_91 wrote: post #2

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

Is the “pretty optimized headless browser” an off the shelf thing, or something custom? Are you using playwright/puppeteer to drive it?

There are plugins for Puppeteer: https://github.com/berstend/puppeteer-extra/tree/master/pack...

Re: The State of Web Scraping 2022

#40

Post by giantrobot »

mschuster91 wrote: post #20
Fantosism wrote: post #12

Earlier quoted context omitted.

I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.

I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers ?!

> But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers?!

There's a whole sneaker collecting subculture. Some buy and wear while others just collect. The big names in sneakers do release limited production models or limited runs of certain color combinations.

Similar to any other collecting subculture.

Post reply on HN