Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

131–140 of 196 posts

Re: Scrape like the big boys

#131

Post by simondotau »

matheusmoreira wrote: post #128
simondotau wrote: post #125

Earlier quoted context omitted.

> You're going to need to explain how scraping publicly available information on a website is theft. Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.

I was responding to a question of whether it was theft, not whether such theft is morally grey or unstoppable in practice.

Yours is a somewhat orthogonal point and one I don’t entirely disagree with.

Re: Scrape like the big boys

#132

Post by emodendroket »

ianbutler wrote: post #118
echelon wrote: post #115

Earlier quoted context omitted.

If data is your competitive advantage or product, then what? Accept that your market no longer exists and that there's no way to stop theft?

You're going to need to explain how scraping publicly available information on a website is theft. If information is your competitive advantage maybe you shouldn't have it on a publicly accessible website, and should instead stick it behind an API with pay tiers and a very clear license regarding what you may do with it as an end user. Note, a simple sign up being required to view a website makes it not publicly avai…

How practical is this really though? Like, imagine you're a newspaper. Unless you're the FT or Wall Street Journal or something like that, nobody is making an account to read an article. They'll just go somewhere else.

Re: Scrape like the big boys

#133

Post by nickff »

matheusmoreira wrote: post #128
simondotau wrote: post #125

Earlier quoted context omitted.

> You're going to need to explain how scraping publicly available information on a website is theft. Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.

>"Do I need to explain that copyright is practically unenforceable in the 21st century?"

This sentence added nothing substantive to your comment, and made it rude; could you please be a bit more polite in the future?

https://news.ycombinator.com/newsguidelines.html

Re: Scrape like the big boys

#134

Post by ushtaritk421 »

zivkovicp wrote: post #49
scarygliders wrote: post #41

Earlier quoted context omitted.

> You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. There's the problem right there. The prices and product data are publicy visible - because there is a target audience of /humans/ for whom the site is designed and intended to be used by. The site is not there to cater for a competitor's scrapers. I don'…

I agree 100%, but it is a fact of life, and sometimes it's better to just minimize the fuzz and focus on the things that matter. Your argument is perfectly valid and applies to offline activities as well (what stops a competitor from walking through the aisles of a Walmart or Costco?), but this is a battle that can't be won, there are too many parasitic actors. It is human nature.

Understanding your competitor's pricing is not "parasitic", it's research. Every company I've ever worked for that sells something online scrapes their competitors in some way (whether with bots or with interns).

Re: Scrape like the big boys

#135

Post by ushtaritk421 »

TeMPOraL wrote: post #71
zo1 wrote: post #69

Earlier quoted context omitted.

From my perspective, the problem is that the data that is offered isn't really "for humans". The data is for convincing the humans to buy/pay or worse, browse and watch ads as a result. But overall, information is one of those goods that has intrinsic properties like no other. It can be copied, infinitely. And we haven't yet figured out the dynamics of how to reason about it, so it feels like we're pretending they're…

You hit the nail on the head. It's hard to have sympathy for site operators complaining about scraping, where almost every site does its best[0] to make using it a time consuming, potentially risky and overall annoying ordeal. Not to mention, information asymmetry is anathema to a well-functioning market, and yet no. 1 reason for fighting bots given in the whole thread here is a desire to maintain that information as…

> Not to mention, information asymmetry is anathema to a well-functioning market, and yet no. 1 reason for fighting bots given in the whole thread here is a desire to maintain that information asymmetry.

This is exactly right.

Re: Scrape like the big boys

#136

Post by matheusmoreira »

nickff wrote: post #133
matheusmoreira wrote: post #128

Earlier quoted context omitted.

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.

> "Do I need to explain that copyright is practically unenforceable in the 21st century?" This sentence added nothing substantive to your comment, and made it rude; could you please be a bit more polite in the future? https://news.ycombinator.com/newsguidelines.html

I was replying to a similar sentence, but it is true that in the end it did nothing but escalate the situation. I apologize and yes, I will try to be more polite.

Re: Scrape like the big boys

#137

Post by ebbp »

scarygliders wrote: post #35
Goronmon wrote: post #25

Earlier quoted context omitted.

If an amateur can do damage to you, then I have some bad news for you... I believe the point wasn't surprise that damage occurred at all, but frustration that damage can occur just out laziness/ignorance rather than malice.

Indeed, that was precisely their point, and "bad news for you" is disingenuous as there are many techniques used by incompetent, or just downright unethical and greedy scraper companies which, no matter how robust the target is, can still give it a major headache. I've witnessed a site being basically DOS'ed due to particularly greedy and aggressive mass scraping attempts.

Precisely this, thank you.

Re: Scrape like the big boys

#138

Post by ushtaritk421 »

scarygliders wrote: post #34
purerandomness wrote: post #19

Earlier quoted context omitted.

Why do you think are they trying to circumvent it? Does your API provide all the information that can be found on the site, or are they scraping because the API is incomplete? We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.

Why would Amazon wish to provide you with easy to access data on their products and prices when you could either be a competitor wishing to undercut those prices, or be a scraper company hired by such a competitor? In what universe is providing such a straightforward way of helping a competitor considered sane business practice?

Markets are competitive and efficient when all parties have full information. If Amazon doesn't want its prices to be known amd finds ways to successfully prevent them from being scrapes, in some sense the state should force it to disclose them via API (or something equivalent)

Re: Scrape like the big boys

#139

Post by throwawaygh »

simondotau wrote: post #125
ianbutler wrote: post #118

Earlier quoted context omitted.

You're going to need to explain how scraping publicly available information on a website is theft. If information is your competitive advantage maybe you shouldn't have it on a publicly accessible website, and should instead stick it behind an API with pay tiers and a very clear license regarding what you may do with it as an end user. Note, a simple sign up being required to view a website makes it not publicly avai…

> You're going to need to explain how scraping publicly available information on a website is theft. Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

OP was talking about price lists. IANAL but AFAIK you can't copyright a list of prices.

Re: Scrape like the big boys

#140

Post by matheusmoreira »

ligerzer0 wrote: post #75
matheusmoreira wrote: post #58

Earlier quoted context omitted.

> Seriously, I hate scrapers. I hate the people who make scrapers. I hate their lack of ethics. Fuck those guys. Not everybody in this space is out to destroy your site. Some of us actively try to put as little load on your site as possible. My scraper puts less load on sites than I do when I browse them normally, I've measured it. Really sucks when we get lumped together with the other abusers and blocked.

Exactly, some of us use scrapers because while we can't go full Richard Stallman, we also don't want to visually sift through ridiculous UI just to look at some basic data/text.

> we also don't want to visually sift through ridiculous UI just to look at some basic data/text

Yeah.

First scraper I ever built was for my school portal. Absolutely atrocious user interface. It got to the point that I seriously hated that site so I built a script to log into it and download my information. I just wanted to see my grades without suffering.

Post reply on HN