Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

111–120 of 145 posts

Re: The State of Web Scraping 2022

#111

Post by samcrawford »

KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Enjoyed reading your bio on your website. Sub 24 hour at Leadville is super impressive! (Coming from someone who has not managed 24 hours at Western States... Yet...)

Re: The State of Web Scraping 2022

#112

Post by autoexec »

fleddr wrote: post #27
joe_91 wrote: post #10

Earlier quoted context omitted.

Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy". You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...

No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…

> Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal.

I suppose it just comes to down to your own morals, but I see nothing at all unethical about scraping a site for personal use provided that it's done gently enough to avoid DoS or disruption. The idea that saving webpages to read later is parasitic or unethical if a website uses robot.txt to discourage commercial scrapers and data-mining goes way too far.

Re: The State of Web Scraping 2022

#113

Post by coverj »

I have been interested in web scraping lately but never really dived too deep. Did anyone have more indepth resources (github projects, blogs, forums, etc) than the tutorials that are basically install beautiful soup and get data from a tag?

Re: The State of Web Scraping 2022

#114

Post by JimBlackwood »

coverj wrote: post #113

I have been interested in web scraping lately but never really dived too deep. Did anyone have more indepth resources (github projects, blogs, forums, etc) than the tutorials that are basically install beautiful soup and get data from a tag?

Genuine question but, what more do you need?

Re: The State of Web Scraping 2022

#115

Post by KieranMac »

samcrawford wrote: post #111
KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Enjoyed reading your bio on your website. Sub 24 hour at Leadville is super impressive! (Coming from someone who has not managed 24 hours at Western States... Yet...)

Leadville is just 45 minutes up the road for me, so I'm kind of cheating!

Re: The State of Web Scraping 2022

#116

Post by JimBlackwood »

blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

If we want to take the unethical route, I’d argue not providing an API (paid or free) is unethical and a nasty business practice.

I work for an ecommerce company and we scrape competitors for price information. Should this automated process using API’s not be okay, we’ll have humans do it. Less efficient for us, more traffic for a competitor. Should they provide a paid API with price information available, I’m sure we’d pay.

Re: The State of Web Scraping 2022

#117

Post by nsonha »

joe_91 wrote: post #2

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

> optimised headless browser with good proxies instead

are you saying you only had problem because you didn't use headless browser before and now with both headless and proxy it generally suffices to not be seen as scrapper?

Re: The State of Web Scraping 2022

#118

Post by nsonha »

RobSm wrote: post #91
akersten wrote: post #78

Earlier quoted context omitted.

> many terms of service prohibit interacting with their website in an automated fashion, Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data! ), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable,…

This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.

are you a lawyer? Your opinion doesn't really mean anything if you still lose the case at the end. By your logic there isn't a clear way to define DDoS either. Sounds like there is though?

Re: The State of Web Scraping 2022

#119

Post by ForHackernews »

digitcatphd wrote: post #72
KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Good take, IMO ethically speaking we should not penalize scrapers themselves but do so based on their use. Scraping Facebook to make a clone of profiles shouldn’t be held to the same scrutiny of scraping Facebook to do an internal analysis of user demographics for research purposes.

Why should either be discouraged?

Re: The State of Web Scraping 2022

#120

Post by nsonha »

mycall wrote: post #33
xrendan wrote: post #24

Earlier quoted context omitted.

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

- Scraping faces to find missing persons: ethical - Scraping photos to create deep learning VQGAN+CLIP art generator: ethical .. we can go on and on, but we should all agree scraping is a useful tool that should never be outlawed.

I don't think that it is outlawed though, at least in practical terms, no one is gonna sue you for scrapping government websites. You really only think about the legal aspect when you do it for commercial gains.

It would be interesting to know if that data can be used in a court case against a government agency though.

Post reply on HN