Live data from Hacker News

GPTBot – OpenAI’s Web Crawler

platform.openai.com

211–220 of 327 posts

Re: GPTBot – OpenAI’s Web Crawler

#211
post #119

Earlier quoted context omitted.

It’s so now they can lobby for anti scraping regulation and hamper any possible catch-up.

How's that gonna work when they need to update their model? Also, how would they compete with companies like FB that have an insane amount of conversational data, or Google, a company that literally indexes the internet?

They're actually paying for access to the AP and other sources now.

Re: GPTBot – OpenAI’s Web Crawler

#213
post #30

Earlier quoted context omitted.

Hoping this is what they’ll use to train future models and deprecate the older ones before the legal cases proceed any further.

The legal cases don't mean anything. The rule of law has all but disappeared from the corporate world. The idea that courts or regulators will be able to control AI is laughable. They are too corrupt, and they are way too slow.

Yeah, here in the USA we haven't figured out Section 230 yet. There is no hope for sensible (or illogical) AI regulation.

Re: GPTBot – OpenAI’s Web Crawler

#214
post #194

Earlier quoted context omitted.

That's just a win-win situation, you're using their services for free because it helps you, they use your interaction to improve the model; the model is still free to use.

There's no win win situation. My content is stolen and given to others. I've lost. Google paid me for traffic via ads, therefore I allowed google to ingest my content. You as a person could read it. I've never given you permission to resell it, and if you did, I'd come after you to pay royalties. The same must apply to openai and other leeches.

> My content is stolen

Physical property is stolen. Information is copied.

Re: GPTBot – OpenAI’s Web Crawler

#215
post #113

Earlier quoted context omitted.

> What do you think why writers and actors have included AI in the reasons of their strike? Because they are about to become obsolete, and they believe that screaming as loudly as they can is going to stop that. Their chances of success are roughly the same as if they were protesting against the law of gravity.

The fun part about being a strong believer of AI and actually understanding its capacities is being able to tell when people are completely blinded by hype. AI will not make writers “obsolete”, that is utterly absurd. Would you say reality TV made tv writers obsolete? No? Oh well. You get what you pay for. That includes what you pay for as a producer…

Call me crazy, but the AI generated Seinfeld brought me more entertainment in the last 6 months than anything Netflix has produced in the last year.

I think they're _very_ worried and rightfully so. I assume it would be very difficult to cancel an AI.

Re: GPTBot – OpenAI’s Web Crawler

#216
post #22
post #5

If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving

You don't have to publish anything on the internet. And when you do, you may limit the allowed audience to just the group of your friends etc. Why publish anything if you worry that someone may consume it?

ChatGPT is not "someone", it's a black box that will ingest everything at his disposal and can't tell you where he gets the information from.

The moral thing to do would be to use opt-in training data.

Re: GPTBot – OpenAI’s Web Crawler

#217

I wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.

I did this to Google for a while only to have my domains listed as malicious. I did not offer any malicious material, just different content for search engines was enough to flag my sites. They also did this to me when I gave google different IP addresses using a split DNS view. This was a while back so maybe they stopped this, I honestly don't know. Now I just give them and most bots a password prompt. Google and most bots can't speak HTTP/2.0 yet. Bing is the exception and I just trust user-agent for them.

    # all nginx virtual sites
    if ($server_protocol != HTTP/2.0) { return 302 https://auth.domain.tld$request_uri; }

    # in auth.domain.tld virtual site
    auth_delay 4s;
    location / { auth_basic "Wamp Wamp"; auth_basic_user_file /dev/shm/.p; }

Re: GPTBot – OpenAI’s Web Crawler

#218

I wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.

There is only one possible move.

Feed them data created by LLMs.

Re: GPTBot – OpenAI’s Web Crawler

#219

I wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.

The training process probably doesn't care and may do unexpected things at scale. You will most likely not be able to outsmart it. It only works to predict the next token, so fake info may even improve its spam detection skills.

Re: GPTBot – OpenAI’s Web Crawler

#220
post #209

Earlier quoted context omitted.

That's of no benefit to me. Quite the contrary.

Why? If it helps people it should be good. Why bother posting something on the public web if not to help people. Sure a large org is receiving some ancillary benefit, but do you feel the same hostility for people working at [large corp] using what you worked on to help them at work? I honestly don't understand the hostility towards llms using public data

>Sure a large org is receiving some ancillary benefit

The large org is receiving the greatest benefit.

Post reply on HN