You literally can't. Data collection at scale is one of the three pillars of the current "AI" hype. It's an issue and it was by design (thanks, three-letters!) and now it will never ever be revoked. Assume all data on the internet is logged in some form and is available to people who shouldn't be able to access it and that those people use it to model things at scale, or sometimes, just store it so they can reinterpret it later. Storing data is cheap. Transmitting data is cheap. MITMing the world's data flow? Priceless.
Ask HN: How do I stop companies from scraping my site?
21–30 of 62 posts
Re: Ask HN: How do I stop companies from scraping my site?
#221. Robots.txt will help somewhat [1]
2. Put it behind a login wall
Re: Ask HN: How do I stop companies from scraping my site?
#23Earlier quoted context omitted.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business. For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatG…
Your apple analogy is completely legal and ethical. The alternative is Monsanto's attempt at DRM with their "terminator" seeds, which many found to be unethical.
Sadly, this is not even clear cut, though of course this is a particularly egregious example.
For instance depending on jurisdiction it might be illegal to sell seeds that can make plants that can produce viable seeds, unless you have a specific authorisation and depending on the species.
Re: Ask HN: How do I stop companies from scraping my site?
#24Commerical web scraping services exist that can scrap any web content. The best you can do is ask nicely in a robots.txt file.
Re: Ask HN: How do I stop companies from scraping my site?
#25Earlier quoted context omitted.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business. For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatG…
> I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business Thereby increasing the amount of apples available to the population, lowering prices, making them available to more people. A net gain for society as a whole. This happen all the time, y…
> A net gain for society as a whole.
Perhaps at first. A race to the bottom usually results in everyone being worse-off though — see red delicious apples.
Re: Ask HN: How do I stop companies from scraping my site?
#26I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs? I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business. For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatG…
The analogy seems fitting. The point is that the content was meant for direct rather than indirect consumption. My conclusion however is different--all are valid uses. Apples are what was sold, not a license for specific use of apples. It would be different if the content was posted on your own site with specific terms of service that prohibited specific uses. I don't think Reddit for example had anything like that in place when the data was scraped.
Re: Ask HN: How do I stop companies from scraping my site?
#27[dead]
Re: Ask HN: How do I stop companies from scraping my site?
#28Earlier quoted context omitted.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business. For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatG…
> I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business Thereby increasing the amount of apples available to the population, lowering prices, making them available to more people. A net gain for society as a whole. This happen all the time, y…
[deleted]
Re: Ask HN: How do I stop companies from scraping my site?
#29Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1] What is not then permitted is to give other people copies, or publish them on your website, or pre…
Publishing a derivative work of something you scraped is not permitted. The question is whether LLMs are derivative works, which has not been settled in courts yet.
Technically correct on the point that it has not been settled in courts yet.
For eg stable diffusion I actually dove a bit deeper on this:
The LAION dataset is -in itself- almost indisputably legal: It contains URLs of images, and tags describing those images[1]. To be sure: _It does not contain actual images_. People have made this mistake before, and LAION quite correctly reply that they don't have them. [2]
Stable diffusion's models are much smaller [3] than the LAION dataset which they use as their source. Again, the models do not contain images.
This is pretty suggestive that SD models should not be considered derivative works. Of course sometimes funny things happen in court, so we can't be sure until a court actually decides.
I don't currently run my own GPT or LLAMA, but those should have similar properties. (in fact, OpenAI appears to claim so.)
[1] https://laion.ai/blog/laion-5b/ (if you look for the files in any of the variations, you'll see it's in the order of terrabytes)
[2] https://www.vice.com/en/article/pkapb7/a-photographer-tried-...
[3] https://huggingface.co/stabilityai/stable-diffusion-xl-base-... : see under "files and versions", it's sd_xl_base_1.0.safetensors. File size is 6.9 GB
Re: Ask HN: How do I stop companies from scraping my site?
#30Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1] What is not then permitted is to give other people copies, or publish them on your website, or pre…
Publishing a derivative work of something you scraped is not permitted. The question is whether LLMs are derivative works, which has not been settled in courts yet.
I do not trust a court to weigh in properly on the deep technical nuances involved.