Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTBot – OpenAI’s Web Crawler

platform.openai.com

Post reply on HN
Print view View original

151–160 of 327 posts

Re: GPTBot – OpenAI’s Web Crawler

#151

Post by rawoke083600 »

planb wrote: post #68
agnosticmantis wrote: post #53

What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…

Just one example: As a software vendor, you probably want OpenAI to index your documentation, so questions about your software can be answered by ChatGPT. Not everybody who creates content is a "content creator" (when did this word get the specific meaning "people who earn money or reputation from creating content").

oh 100% The hoops our current generation has jumped through(including me) to make abs sure google can index your site ! I think for some mental-models or product-segments (like software vendor example) it's definitely essential to be part of the new paradigm of information-access

Re: GPTBot – OpenAI’s Web Crawler

#152

Post by rawoke083600 »

ducktective wrote: post #63

Friendship ended with SEO. Now LEO [1] is my best friend. [1]: LLM Engine Optimization

I can also (depending on industry) see a dual model (freemium vs paid):

If (generally) more data is better. As a site owner I might be happy to give free access to most of my public data and urls. You be at the mercy of my sites unique formatting and hiccups

But for some fee, I might be happy to provide api-like data access to some of my historic data that are more rich with information and promised some sort of format-encoding (AI-JSON ?)

The economics I think are still currently be discovered.

Re: GPTBot – OpenAI’s Web Crawler

#153

Post by Fnoord »

rawoke083600 wrote: post #141

Man what a time we live in :) It's like history is being written (ok compiled & backpropagated) right under our feet ! I can see a bots.txt entry in the near future that discern the site's data-usage for bots vs humans User-agent-class: AI Data-Policy-Allow: /news/* /articles/* Data-Policy-Deny: */comments

Indeed!

Although no human is going to read robots.txt or bots.txt

It'll end up as a small section in the EULA of the website which nobody reads:

> Before you click the 'reply' button please be aware we are allowing AI to crawl our comment section for training. Thank you for your consideration.

There's a little problem though:

1) Websites don't have an incentive to inform their users about this, and websites don't have an incentive to allow AI to crawl their content unless they get something back from it (e.g. payment). From this PoV, its time for OpenAI to start paying.

2) The competition (China, Russia) doesn't care about bots.txt or robots.txt and will just crawl whatever the hell they can.

Re: GPTBot – OpenAI’s Web Crawler

#154

Post by gumballindie »

uallo wrote: post #39

Yet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death. Luckily, I already block the IP address they're using as it has been used for (other?) malicious bots before. [1] In my case, it relies on third-party APIs that are heavily rate limited. Any bot ignoring rate limitation measures will effectively (D)DOS my serv…

One option is to completely ban openai’s crawler ip addresses. They steal content without credit anyway - as most ai companies do - so there’s no benefit in allowing them access.

Re: GPTBot – OpenAI’s Web Crawler

#155

Post by gumballindie »

ketzo wrote: post #16
wg0 wrote: post #5

If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving

Counterpoint (not just to be annoying — I think you pose a very interesting unanswered question): If I read your hobby website about photography and use it to take 1% better pictures, do I owe you 1% of what my clients pay me? I think that probably most people would say no, assuming you could even determine that 1% in a way that both parties agreed was fair. I think generally, we have an understanding that some stuff…

An ai is not a “you”. It’s a peace ld software that steals data, rinses it, and monetises it. There is no human like learning.

Re: GPTBot – OpenAI’s Web Crawler

#156

Post by sangnoir »

Namidairo wrote: post #34

I wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.

Randomly filter a subset of responses to OpenAI through the smallest, barely functional LLM one can find, naturally.

Re: GPTBot – OpenAI’s Web Crawler

#157

Post by zarzavat »

gmerc wrote: post #119
Zaheer wrote: post #4

Nice of them to respect crawling after they've already trained their model. Presumably these headers don't affect any pages they've already crawled to train GPT(?)

It’s so now they can lobby for anti scraping regulation and hamper any possible catch-up.

That would be a hilariously bad idea for them. Their business is based on fair use. The only way to enforce restrictions against scraping is through copyright law because obviously you can run the spidering code from any jurisdiction you want, so any law that says “thou shall not scrape” is toothless unless it acts through copyright. Any workable restrictions against using scraped data would also make ChatGPT illegal too.

Re: GPTBot – OpenAI’s Web Crawler

#158

Post by gumballindie »

agnosticmantis wrote: post #53

What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…

There’s little to no incentive. The issue we cant seem to be able to prevent openai from stealing content.

Re: GPTBot – OpenAI’s Web Crawler

#159

Post by Fnoord »

planb wrote: post #68
agnosticmantis wrote: post #53

What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…

Just one example: As a software vendor, you probably want OpenAI to index your documentation, so questions about your software can be answered by ChatGPT. Not everybody who creates content is a "content creator" (when did this word get the specific meaning "people who earn money or reputation from creating content").

The software vendor documentation, for sure.

But the book writer who wrote a detailed, expert book on how to deal with the software ("Photoshop for Dummies"?). OpenAI might be seen as a competitor.

A government would be easier to say all their data isn't allowed to be crawled, so they can sue later or just say no later on when they figure something classified was in there, or simply when they change their mind.

I believe the default response should be 'no, we'll look into it' for anyone, and then carefully let legal take a look at it (gonna be expensive). For the software vendor, too. Although their crown jewels are likely the source code to their product(s).

Re: GPTBot – OpenAI’s Web Crawler

#160

Post by dmitrysergeyev »

Namidairo wrote: post #34

I wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.

They could in theory combat it by comparing results with a second crawler that uses a different User Agent.
Post reply on HN