Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

21–30 of 145 posts

Re: The State of Web Scraping 2022

#21

Post by usbqk »

blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think if you make intangible things public you shouldn’t consider them to be only yours anymore.

Re: The State of Web Scraping 2022

#23

Post by zffr »

KieranMac wrote: post #18
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

There are pro-social and anti-social uses of web scraping. If you have ever used Kayak or any other price discovery or price comparison website, you've relied on web scraping to provide you a service.

Also google or any other search engine

Re: The State of Web Scraping 2022

#24

Post by xrendan »

blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping)

- Scraping public information from government websites to do analysis: ethical, it's the public's data

- Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remittance process: ethical

- Scraping faces to build a surveillance-tech company: disgusting

- Scraping your own website because your internal processes are so broken you can't get it any other way: ethical

- Scraping to just copy someone's data they worked hard to generate to go and resell: unethical

Re: The State of Web Scraping 2022

#25

Post by NDizzle »

I still have a daily job running a web scraper I first wrote with Scrapy back in 2017. I think I've had to update it 3 times over the years for changes to the site and web standards.

Good old government sites - rarely change!

Re: The State of Web Scraping 2022

#26

Post by pjc50 »

mschuster91 wrote: post #20
Fantosism wrote: post #12

Earlier quoted context omitted.

I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.

I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers ?!

Same as NFTs: hype and resale value. At least you can wear the sneakers once you've stopped flipping them.

Re: The State of Web Scraping 2022

#27

Post by fleddr »

joe_91 wrote: post #10
Terry_Roll wrote: post #8

Earlier quoted context omitted.

You dont even need to do that, go overt plain sight in yer face and call yourself a search engine!

Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy". You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...

No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial.

Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal.

"It's public" is a legal defense, not an ethical one. It's public for readers, not for scrapers. It's public within the original context of the website, which may include monetization.

Photographing every page of a book and then reading it that way may be legally allowed, but it's still unethical.

I have somebody in our neighborhood that instead of paying for private trash, takes tiny bags of his private trash to the park and dumps it into the public trash cans.

Legal? Yes. Parasitic behavior? Also yes.

Re: The State of Web Scraping 2022

#28

Post by lambic »

xrendan wrote: post #24
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

The first one here is important. Despite the open data movement pressuring governments to provide their data in easily consumable forms, a lot of government organizations are still unable or unwilling to do so.

Political advocacy orgs rely a lot on scraping to collect political representative data that isn't available through any other means.

Re: The State of Web Scraping 2022

#29

Post by mycall »

ok_coo wrote: post #22

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/

I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org

Re: The State of Web Scraping 2022

#30

Post by nichch »

mschuster91 wrote: post #20
Fantosism wrote: post #12

Earlier quoted context omitted.

I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.

I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers ?!

Economic value and actual usage.
Post reply on HN