Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

71–80 of 145 posts

Re: The State of Web Scraping 2022

#71

Post by danbmil99 »

fleddr wrote: post #27
joe_91 wrote: post #10

Earlier quoted context omitted.

Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy". You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...

No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…

What you fail to acknowledge is that Bing Google etcetera have an effective monopoly on search. They can afford to respect robots.txt because everyone wants them to scrape their site.

The first mover advantage is so huge in this case that without allowing scraping, it's hard to understand how anyone could ever compete with these monoliths.

Re: The State of Web Scraping 2022

#72

Post by digitcatphd »

KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Good take, IMO ethically speaking we should not penalize scrapers themselves but do so based on their use.

Scraping Facebook to make a clone of profiles shouldn’t be held to the same scrutiny of scraping Facebook to do an internal analysis of user demographics for research purposes.

Re: The State of Web Scraping 2022

#73

Post by edmundsauto »

fareesh wrote: post #67

My toolbox of choice for web scraping is either Nokogiri or puppeteer Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?

One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.

Re: The State of Web Scraping 2022

#74

Post by mschuster91 »

nichch wrote: post #30
mschuster91 wrote: post #20

Earlier quoted context omitted.

I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers ?!

Economic value and actual usage.

What economic value (other than hoping for the value to increase, aka tulip mania) can I derive out of ultra-rare sneakers? What usage that goes above "it looks cooler than an unbranded, otherwise identical sneaker"?

For me, this kind of product is part of the "bullshit economy" - similar to "bullshit jobs", this kind of product has no reason to exist other than vanity, as almost all of these "collectibles" won't ever be used. We are using up valuable, finite resources to create and distribute this kind of useless "bullshit product", we are using up valuable human time and IT resources on developing websites capable to resist (D)DoS attacks and on developing snipers to bypass the anti-bot technologies employed by the shops, and we are creating a lot of demand for all kinds of sneaker-related crime - and there's a lot of that: theft and robberies from stores, theft and robberies in the supply chain, ebay/classifieds scams, credit card fraud, robberies on broad daylight [1].

Seriously, fuck all that shit. No one needs hundreds of dollars worth of sneakers that only incentivize crime and bullshit.

[1]: https://www.google.com/search?q=man+robbed+because+of+sneake...

Re: The State of Web Scraping 2022

#75

Post by fareesh »

edmundsauto wrote: post #73
fareesh wrote: post #67

My toolbox of choice for web scraping is either Nokogiri or puppeteer Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?

One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.

Ah interesting, I end up doing this manually, i.e. File.write followed by what I want to scrape

Re: The State of Web Scraping 2022

#76

Post by temp8964 »

joe_91 wrote: post #2

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

I think it will eventually goes to like stock trading. If you have a good strategy, you don't want to share with the world, because it will render your strategy useless.

Re: The State of Web Scraping 2022

#77

Post by tommek4077 »

bobblywobbles wrote: post #61

Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.

Because those terms are the law and cant be ignored in almost all the rest of the world...

Re: The State of Web Scraping 2022

#78

Post by akersten »

bobblywobbles wrote: post #61

Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.

> many terms of service prohibit interacting with their website in an automated fashion,

Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data!), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable, or do I have your permission to let Chrome automate that for me?

Re: The State of Web Scraping 2022

#79

Post by akersten »

xrendan wrote: post #24
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

So it's really not about the "scraping" here, it's about the kind of business you're building. I don't think any of your definitions change if you simply employed people to check the websites instead of scripts.

Re: The State of Web Scraping 2022

#80

Post by Arnavion »

mymllnthaccount wrote: post #60
bluedays wrote: post #51

Earlier quoted context omitted.

Wouldn’t a queuing system be more fair?

By queuing, do you mean first come first serve? No, that causes the problem. That encourages people to use bots to be the first one to purchase the moment the inventory is released. I don't understand how a random raffle would ever not be fair (with the assumption that one person gets only one entry)

That assumption doesn't seem like it'd hold. You'd just replace bot services with package forwarding services that can generate unique PO Box numbers or whatever.
Post reply on HN