Just a random thought: While they are already earning money by scraping data, would it not be nice if they pay the site owners a certain amount of the money they earn?
God no, can you imagine what the web will look like if every grifter can get paid just for existing?
GPTBot – OpenAI’s Web Crawler
71–80 of 327 posts
Re: GPTBot – OpenAI’s Web Crawler
#72Re: GPTBot – OpenAI’s Web Crawler
#73Nice of them to respect crawling after they've already trained their model. Presumably these headers don't affect any pages they've already crawled to train GPT(?)
GPT-4 finished training in August 2022, before the release of ChatGPT. If they had announced this sooner hardly anyone on the internet would have noticed. Props to them for adding it now.
Re: GPTBot – OpenAI’s Web Crawler
#74Is there any argument in favor of commercial websites allowing GPTBot to crawl them? It's not like Google where allowing crawling brings you traffic. In fact, it's pretty much the opposite.
Re: GPTBot – OpenAI’s Web Crawler
#75What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…
Re: GPTBot – OpenAI’s Web Crawler
#76Just a random thought: While they are already earning money by scraping data, would it not be nice if they pay the site owners a certain amount of the money they earn?
God no, can you imagine what the web will look like if every grifter can get paid just for existing?
Re: GPTBot – OpenAI’s Web Crawler
#77Earlier quoted context omitted.
quick back-of-envelope/googling: openai is worth $29,000,000.00 you contributed 0.00000000001 punches numbers in calculator thus the value of your free credits is 0.001 cents. minus any accounting fees.
>openai is worth $29,000,000.00 You might have missed a few zeros.
Re: GPTBot – OpenAI’s Web Crawler
#78Earlier quoted context omitted.
Hoping this is what they’ll use to train future models and deprecate the older ones before the legal cases proceed any further.
The legal cases don't mean anything. The rule of law has all but disappeared from the corporate world. The idea that courts or regulators will be able to control AI is laughable. They are too corrupt, and they are way too slow.
Courts are more deliberate than you would like — no denying that. But this is a feature not a flaw. It may be that damage will be done by then. Perhaps irreversible. But I would like to think if there is a will there is a way and that if things are terrible enough the governments will be bold in their responses.
Re: GPTBot – OpenAI’s Web Crawler
#79Friendship ended with SEO. Now LEO [1] is my best friend. [1]: LLM Engine Optimization
What's the end goal? To teach it some very specific information, like about your company?
Re: GPTBot – OpenAI’s Web Crawler
#80Just a random thought: While they are already earning money by scraping data, would it not be nice if they pay the site owners a certain amount of the money they earn?
If a human read a website and profited from the knowledge obtained, I don’t think we’d expect them to pay royalties to the site owner.