Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I stop companies from scraping my site?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Ask HN: How do I stop companies from scraping my site?

#41

Post by pixl97 »

richbell wrote: post #25
IMTDb wrote: post #18

Earlier quoted context omitted.

> I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business Thereby increasing the amount of apples available to the population, lowering prices, making them available to more people. A net gain for society as a whole. This happen all the time, y…

> A net gain for society as a whole. Perhaps at first. A race to the bottom usually results in everyone being worse-off though — see red delicious apples.

It turns out people want pretty things, not 'good' things. It is very difficult to prevent this from occurring without other nasty side effects.

Re: Ask HN: How do I stop companies from scraping my site?

#42

Post by pixl97 »

BlueTemplar wrote: post #23
satyrnein wrote: post #16

Earlier quoted context omitted.

Your apple analogy is completely legal and ethical. The alternative is Monsanto's attempt at DRM with their "terminator" seeds, which many found to be unethical.

Sadly, this is not even clear cut, though of course this is a particularly egregious example. For instance depending on jurisdiction it might be illegal to sell seeds that can make plants that can produce viable seeds, unless you have a specific authorisation and depending on the species.

Legality has nothing to do with ethical. I by force could make it a law that you have to pay me to eat, most reasonable people would consider that an affront to ethical behavior.

Re: Ask HN: How do I stop companies from scraping my site?

#43

Post by Slavaqua »

throwawaydghvhv wrote: post #36

On my site I don't need SEO, so I sprinkled it with invisible links leading to an endless Markov chain generated walls of garbage text. The robots seem to love it! (If you go similar route, don't forget rate limiting)

I don't understand your solution. The actual content is still being scraped, is it not?

Re: Ask HN: How do I stop companies from scraping my site?

#44

Post by Kim_Bruning »

satyrnein wrote: post #35
Kim_Bruning wrote: post #29

Earlier quoted context omitted.

Technically correct on the point that it has not been settled in courts yet. For eg stable diffusion I actually dove a bit deeper on this: The LAION dataset is -in itself- almost indisputably legal: It contains URLs of images, and tags describing those images[1]. To be sure: _It does not contain actual images_. People have made this mistake before, and LAION quite correctly reply that they don't have them. [2] Stable…

Thanks for the links! To me (merely an amateur observer), it is not perfectly obvious that the LAION dataset is legal, because I don't think "we don't have your original images anymore, we only kept the results of applying an algorithm to them" is a universal defense. If the algorithm is "resize to 640×480" you are probably still infringing, but if it's "calculate md5 checksum" you're probably not. I don't know where…

Heh, it does in fact calculate the MD5 checksum!

In total, the fields in the LAION dataset are:

  image_path
  caption
  NSFW
  similarity
  LICENSE
  url
  key
  status
  error_message
  width
  height
  original width
  original height
  exif
  md5
https://huggingface.co/datasets/laion/laion2b-multi-vit-l-14...

Note that LAION is a metadata dataset, it's not an image generation model itself. (But it has been used to train one)

Re: Ask HN: How do I stop companies from scraping my site?

#45

Post by barrysteve »

hermannj314 wrote: post #12

I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs? I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.

I sell apples at the market. Someone took my apple for free and made a pie out of it. They said it was not theft, because my market stall was in the public.

I sold an apple with DRM. One person bought my apple and gave it to a Chinese hacker, whom stripped the DRM and duplicated my apple for everyone, for free. They said it was not theft.

I grew an apple and put it on a cloud provider's server. I sold access to many duplicated apples, until a technology update forced me to change providers. I found out I could not take back my apples from the cloud providet without paying exorbitant fees. They said it was not theft.

I made an encrypted apple, I sold it on an app store. The app store banned me and my apple, the virus software flagged my apple, the TPM chip stopped my apple from being played, the government cracked open the encryption and duplicated my apple for national security purposes. They said it was not theft.

The only way to sell apples, is through perfectly-black, 'black boxes'. Putting apples on computers, gets them stolen. End of story.

It's been forty years now. There's nothing to be 'unsure about'.

'Benefit of the doubt' for intellectual property theft is deader than a dodo.

Re: Ask HN: How do I stop companies from scraping my site?

#46

Post by Slavaqua »

Kim_Bruning wrote: post #9

Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1] What is not then permitted is to give other people copies, or publish them on your website, or pre…

Seeing as you are knowledgeable on the subject, may be you know if there is a licence, like creative commons, that will allow all the usual intelectual property uses except in AI training?

Or may be an old licence that is so strict that it will protect from AI shenanigans by default?

Re: Ask HN: How do I stop companies from scraping my site?

#47

Post by Slavaqua »

microflash wrote: post #10

I wrote about this a while ago [1]. Unfortunately, with robots.txt, you're at the mercy of crawlers. They may respect it or ignore it altogether. You can block IP addresses but many crawlers may not even use static IP addresses. You can go to extremes and put your content behind a login, as others have suggested. But that would also create friction for your intended audience. [1]: https://www.naiyerasif.com/post/2023…

It sounds like a loosing battle to manually keep track of all bots and their deployment IPs that generate datasets that LLM's might use or start using in the future. There must be a legal solution, a licence that forbids the use of content for training without explicit permission from the author.

Re: Ask HN: How do I stop companies from scraping my site?

#48

Post by Slavaqua »

quickthrower2 wrote: post #7

You can gate it behind a login. Even a simple self made captcha (what is 2 + 7?) to reveal the content would probably stop LLMs. But hurt seo so you have to not rely on that. Do a medium and show a paragraph first then the login/captcha to continue.

Wouldn't it severely affect SEO? I don't know, but I assume, that scrambling article in to a word soup and exposing it to crawlers would not solve the SEO problem. Would it?

Re: Ask HN: How do I stop companies from scraping my site?

#49

Post by xyzal »

Slavaqua wrote: post #43
throwawaydghvhv wrote: post #36

On my site I don't need SEO, so I sprinkled it with invisible links leading to an endless Markov chain generated walls of garbage text. The robots seem to love it! (If you go similar route, don't forget rate limiting)

I don't understand your solution. The actual content is still being scraped, is it not?

I think they are interested in rendering the corpus of text from their site unsuitable for training LLMs by stuffing it with nonsense more than preventing scraping.
Post reply on HN