Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTBot – OpenAI’s Web Crawler

platform.openai.com

Post reply on HN
Print view View original

321–327 of 327 posts

Re: GPTBot – OpenAI’s Web Crawler

#321

Post by terrytanys »

evandale wrote: post #215
scrollaway wrote: post #199

Earlier quoted context omitted.

The fun part about being a strong believer of AI and actually understanding its capacities is being able to tell when people are completely blinded by hype. AI will not make writers “obsolete”, that is utterly absurd. Would you say reality TV made tv writers obsolete? No? Oh well. You get what you pay for. That includes what you pay for as a producer…

Call me crazy, but the AI generated Seinfeld brought me more entertainment in the last 6 months than anything Netflix has produced in the last year. I think they're _very_ worried and rightfully so. I assume it would be very difficult to cancel an AI.

Maybe you can elaborate more on why you are more entertained, then some producers from Netflix can take note and improve.

Re: GPTBot – OpenAI’s Web Crawler

#322

Post by terrytanys »

planb wrote: post #68
agnosticmantis wrote: post #53

What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…

Just one example: As a software vendor, you probably want OpenAI to index your documentation, so questions about your software can be answered by ChatGPT. Not everybody who creates content is a "content creator" (when did this word get the specific meaning "people who earn money or reputation from creating content").

That's good point. ChatGPT itself is very valueable. The problem is for the people who live off creating content.

Re: GPTBot – OpenAI’s Web Crawler

#323

Post by terrytanys »

JimDabell wrote: post #181
agnosticmantis wrote: post #53

What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…

> What’s the incentive for people to allow the crawler at all? So that LLMs can learn from it? Profit is not the only thing that motivates people. I’ve spent years contributing to Stack Overflow to help people solve their problems, with the understanding that they had an open data policy and anybody could access the data dump easily to build things with it. It pisses me off that they are now trying to lock that infor…

Thank you for your contribution! I think that has to be a strategic decision. If everyone start using ChatGPT for everything, what's the value now for SO? From their perspective, they wouldn't sit and watch it happens. And I would add citation is a big deal.

Re: GPTBot – OpenAI’s Web Crawler

#324

Post by terrytanys »

kasdi wrote: post #293
yosito wrote: post #67

Is there any argument in favor of commercial websites allowing GPTBot to crawl them? It's not like Google where allowing crawling brings you traffic. In fact, it's pretty much the opposite.

Think about an AI as a personal assistant - for the whole world. Would you want the assistant to know about your business? In most cases I do think so. It could mention you in a conversation, the analog to you appearing in Google Search results. And maybe even better, provide the necessary context to generate more real customers for your business. You don’t want traffic to your website, you want customers to your bus…

OpenAI have to know they are going into consumer business not just an AI research company.

Re: GPTBot – OpenAI’s Web Crawler

#325

Post by literalAardvark »

callalex wrote: post #268
literalAardvark wrote: post #104

Earlier quoted context omitted.

Chatgpt 4 provides pretty good citations on request.

It also keeps getting caught just blatantly making up citations that look good.

That's mostly a problem with GPT3, not GPT4. I'm not saying it doesn't make some of them up, but I've had great research experiences with it.

It's true that after you use the bot to fetch you the papers, you do still need to read them... but given what a dramatic difference there is between GPT3 and 4 I'd say this is a problem that will be utterly annihilated before most people even hear it exists.

Re: GPTBot – OpenAI’s Web Crawler

#326

Post by keymon-o »

awb wrote: post #246
keymon-o wrote: post #127

Earlier quoted context omitted.

Nobody is doubting that accessible and correct information is good for humanity, I am questioning how will that affect knowledge/content providers. I repeat, what will be the motivation of an individual to share or provide valuable information if you decrease or eliminate any control of where and how that information appears?

> I repeat, what will be the motivation of an individual to share or provide valuable information if you decrease or eliminate any control of where and how that information appears? Forum users don’t seem to mind. Reddit, HN, Twitter, Facebook, etc. are all examples of users freely providing valuable content without expectation or control. I suppose it’s also not too different from a listener summarizing a speech. Wh…

Very late to reply. Still you take part in a community, you have an identity and you get likes, karma, followers and credibility. Motivation is present just like in monetized systems and you choose where it appears and its related to your account.

In the meantime, I did relize I might have overblown consequences of what a product collecting and summarizing knowledge might cause.

Re: GPTBot – OpenAI’s Web Crawler

#327

Post by fulmicoton »

insanitybit wrote: post #237
gvkhna wrote: post #233

Earlier quoted context omitted.

I’m working on this problem (heavy user of chatgpt too). What kinds of libraries do you use it for that are out of date. I could hopefully get you into the beta with it having better responses for those libs. Please email me gaurav@gvkhna.com

Rust libraries as well as Hashicorp Nomad (has changed a lot since ChatGPT's last training point). Also QuickWit is totally unknown to ChatGPT.

It has information from 2021. ChatGPT presents Quickwit as follows:

As of my last knowledge update in September 2021, Quickwit is an open-source search engine infrastructure that is designed for building and deploying search solutions quickly and efficiently. It focuses on providing fast and scalable full-text search capabilities for applications and websites. Quickwit is built on top of the Rust programming language and leverages technologies like the tantivy search engine library.

Post reply on HN