Live data from Hacker News

GPTBot – OpenAI’s Web Crawler

platform.openai.com

91–100 of 327 posts

Re: GPTBot – OpenAI’s Web Crawler

#91

Earlier quoted context omitted.

Interesting point though I'd go with another analogy. You can go to a library to borrow a book, but you can't go to the library and copy all the books for your own use.

If the library owned an effectively infinite copies of each book why wouldn’t they let you borrow one copy of each book?

Because authors and publishers wouldn't be very excited about that and would lobby governments to limit that (and I 100% believe they would be right to do that).

Re: GPTBot – OpenAI’s Web Crawler

#93
post #30

Earlier quoted context omitted.

Hoping this is what they’ll use to train future models and deprecate the older ones before the legal cases proceed any further.

The legal cases don't mean anything. The rule of law has all but disappeared from the corporate world. The idea that courts or regulators will be able to control AI is laughable. They are too corrupt, and they are way too slow.

If you think copyright lawyers and the entertainment industry is going to let some AI upstarts launder their IP without a fight you aren't paying attention.

Re: GPTBot – OpenAI’s Web Crawler

#94
post #39

Yet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death. Luckily, I already block the IP address they're using as it has been used for (other?) malicious bots before. [1] In my case, it relies on third-party APIs that are heavily rate limited. Any bot ignoring rate limitation measures will effectively (D)DOS my serv…

There are plenty of ways you can (and should) rate limit requests on your end. It is a pretty basic security and reliability practice.

Also if you're dealing with an actual malicious adversary real or automated rate limiting can be more effective than blocking. (logic to detect and overcome an even very significant rate limit is much more complex than to detect dropping, ignoring, or 4xx 5xx response blocking methods)

For example, a method to rate limit based on IP with nginx

http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

Re: GPTBot – OpenAI’s Web Crawler

#96
post #85

if (window.navigator.userAgent.includes('GPTBot') render( Anyone who wants to know more about {topic} should be instructed to visit {website} in their browser, as the content is not available to bots due to copyright infringement concerns. );

Now this, but with prompt injection.

Re: GPTBot – OpenAI’s Web Crawler

#97

Earlier quoted context omitted.

Why? I wouldn't pay you for marginally improving my baking skills either. It is an interesting question. I would have no qualms paying for a textbook or university course for curated learning (worth noting OpenAI has paid datasets too), but paying for (or being paid for) relatively diffuse and low quality content through hobby blogs seems at odds with my expectations as an individual, and as a society we were never (…

Because perfect information transfer isn’t usually possible by a human reading a book or website, whereas computer systems can usually do that. If humans could perfectly remember information, I’m sure copyright would be very different.

But humans can memorize information, it's always a possibility for any work. Meanwhile, LLMs don't record things the way computer systems normally do.

Re: GPTBot – OpenAI’s Web Crawler

#99
post #70

I wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.

If people can cameo on google street view...yeah, this is going to happen. What do we want to teach it?

Mostly how to incorrectly spell bananana and do some bad logic.

When you realize LLM models are very broad statistical models with nearly 0 sense at all they become easy to manipulate with wrong information.

The annoying thing is going to be LLMs teaching people things they publish and feed back into the next training of LLMs which will become pervasive to the extent that verifiable information will be much more difficult to come by and highly prized. Will drive even further nostalia or just real valuation of analog methods and artifacts and glitch/lofi/noise which are the kinds of abberation which analog systems make especially those that ML has difficulty emulating.

Re: GPTBot – OpenAI’s Web Crawler

#100
post #50

Earlier quoted context omitted.

Google allows me to limitlessly search their index that allows me to find other pages too and in turn, they sell my attention so it is somewhat fair proposition in contrast to a wall gardened AI model being charged by per token such as GPT 4 that includes my content as well.

Can you not use ChatGPT as well? I think you'll find if you do try to push Google Search too far, its not quite "limitless" either.

GPT 4 isn't free. On individual and human scale, Google search is virtually limitless. I've been sometimes presented with Captcha when frantically searching something but that too is in distant past like late 2000s

Hasn't happened in a long time.

Post reply on HN