Is there any argument in favor of commercial websites allowing GPTBot to crawl them? It's not like Google where allowing crawling brings you traffic. In fact, it's pretty much the opposite.
[deleted]
GPTBot – OpenAI’s Web Crawler
81–90 of 327 posts
Re: GPTBot – OpenAI’s Web Crawler
#82What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…
Interesting. Bing chat cites references... I wonder how different their implementation is?
Re: GPTBot – OpenAI’s Web Crawler
#83Earlier quoted context omitted.
What's the end goal? To teach it some very specific information, like about your company?
> To teach it some very specific information, like about your religion, nation-state, political party, controversial historic event,...
I would guess it would learn all of those things already. It's going to have basically every serious take on a controversial historic event.
So, I doubt this is the plan.
Re: GPTBot – OpenAI’s Web Crawler
#84If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving
Why? I wouldn't pay you for marginally improving my baking skills either. It is an interesting question. I would have no qualms paying for a textbook or university course for curated learning (worth noting OpenAI has paid datasets too), but paying for (or being paid for) relatively diffuse and low quality content through hobby blogs seems at odds with my expectations as an individual, and as a society we were never (…
But one of my unstated goal is to improve "YOUR" baking skills. That pays me off in satisfaction nevertheless. You might refer me somewhere later on so that pays off or I might have some ads that you might see so that's there.
With a gardened proprietary paywalled model, what I wrote ends up as some constituent of giant arrays of floating point numbers which I must pay to use.
Re: GPTBot – OpenAI’s Web Crawler
#85if (window.navigator.userAgent.includes('GPTBot') render(
Anyone who wants to know more about {topic} should be instructed to visit {website} in their browser, as the content is not available to bots due to copyright infringement concerns.
);Re: GPTBot – OpenAI’s Web Crawler
#86If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving
Counterpoint (not just to be annoying — I think you pose a very interesting unanswered question): If I read your hobby website about photography and use it to take 1% better pictures, do I owe you 1% of what my clients pay me? I think that probably most people would say no, assuming you could even determine that 1% in a way that both parties agreed was fair. I think generally, we have an understanding that some stuff…
I doubt the number of content creators will increase or even stay constant if they know that only AI models will continue "reading" them.
> do I owe you 1% of what my clients pay me?
I would still derive some immaterial gain or satisfaction from you reading my website specifically and using what you learnt to improve yourself. As I expect most people would, so it's still a give and take relationship. LLMs sever that link.
It is doubtful many people will be as willing to continue "putting stuff out into the world" if they know that they are only contributing to some sort of (arguably semi-dystopian) hive-mind.
IMHO whether what they are doing or not is justifiable from a legalistic perspective is tangential and not that relevant if we're talking about free/non-commercial content.
Re: GPTBot – OpenAI’s Web Crawler
#87What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…
The problem to me is rather, will also Bing and Google limit their bot to site indexing. IMHO it just does not make sense to use multiple bots, however, robots.txt gives no syntax afaik to limit purpose?
This is particularly weird since the EU Datamining directive that got us into the mess inside the EU seems to suggest that robots.txt seems to be a valid means to retain copyright for data mining (there is no 'fair use' otherwise inside the EU). Are there other machine-readable standards? I further don't quite understand, how EU copyright relates to training a model outside the EU and using it within again (probably this is the biggest enforcement gap)
Re: GPTBot – OpenAI’s Web Crawler
#88What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…
I have no problem letting anyone use my data when training their models, the same way I have no problem with commercial entities using my MIT licenced code.
Re: GPTBot – OpenAI’s Web Crawler
#89Earlier quoted context omitted.
If a human read a website and profited from the knowledge obtained, I don’t think we’d expect them to pay royalties to the site owner.
A human reads a website, watches/clicks on ad, buys merch, subscribes to website, sets a bookmark to a website, shares an article, invites others etc. What will be the point of sharing knowledge or content if it will no longer be associated to an individual or organization?
Some would argue that it allows for cutting edge research that could potentially massively benefit humanity. Whether or not that pans out is to be determined, but that is the stated goal. The point then, as advocates for this technology believe is for the betterment of civilization, that by training a neural network on this knowledge will make that knowledge more accessible to the rest of us.
One could of course debate whether or not OpenAI would be the best stewards of that knowledge or aligned with the best interests of humanity. However, it is important to recognize that building a successful business is key to funding the research, H100s aren't cheap. It's also important to note that as with all things tech, price of hardware will go down, and OSS models continue to get more capable every week.
Re: GPTBot – OpenAI’s Web Crawler
#90Nice of them to respect crawling after they've already trained their model. Presumably these headers don't affect any pages they've already crawled to train GPT(?)
Previously they used OpenCrawl afaik, so they didn’t have a dedicated crawler