Earlier quoted context omitted.
How can doing this cannot affect your SEO?
Serve different content to non search engine bots
I often read that doing that was a big no-no as Google would flag your website as cheating. But that may be a SEO legend.
Ask HN: How do I stop companies from scraping my site?
11–20 of 62 posts
Re: Ask HN: How do I stop companies from scraping my site?
#12I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs?
I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
Re: Ask HN: How do I stop companies from scraping my site?
#13If it's worth that much, just put it behind a login.
Re: Ask HN: How do I stop companies from scraping my site?
#14Make some of your content, which is invisible to normal users, so exceptionally toxic to the bots that they will begin to avoid your site by choice.
How can doing this cannot affect your SEO?
If SEO's a concern, then there's nothing you can do about the LLM bit. Either put the website behind a login, or deal with robots sucking everything up.
Pre-LLMs, there were already many websites showing scraped versions of other people's sites, and ranking higher in search results.
Re: Ask HN: How do I stop companies from scraping my site?
#15I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs? I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business.
For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatGPT can customize it's response based on my unique situation whereas the Medium article is fixed, but it's still unfair to the author of the Medium article that ChatGPT was trained on, because without their effort, none of this would have been possible. And yet, because I am no longer generating as revenue for Medium, the original authors are no longer being compensated for the value they provided me.
I personally believe that LLM hosts should be required to pay for training data. With RAG, I think it makes sense to charge per-query, but for base models, content authors would probably need to organize into larger groups to facilitate payment.
Re: Ask HN: How do I stop companies from scraping my site?
#16I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs? I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business. For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatG…
Your apple analogy is completely legal and ethical. The alternative is Monsanto's attempt at DRM with their "terminator" seeds, which many found to be unethical.
Re: Ask HN: How do I stop companies from scraping my site?
#17[deleted]
Re: Ask HN: How do I stop companies from scraping my site?
#18I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs? I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business. For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatG…
> I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business
Thereby increasing the amount of apples available to the population, lowering prices, making them available to more people. A net gain for society as a whole.
This happen all the time, you benefit from it all the time. You only have a problem with it because for once this system is not working in your direct favour.
Re: Ask HN: How do I stop companies from scraping my site?
#19Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1] What is not then permitted is to give other people copies, or publish them on your website, or pre…
Publishing a derivative work of something you scraped is not permitted. The question is whether LLMs are derivative works, which has not been settled in courts yet.
Re: Ask HN: How do I stop companies from scraping my site?
#20You can take it off the web. Frankly these concerns confuse me. You published so people can find value, no?