Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

251–260 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#251

Post by Dylan16807 »

nicolas_17 wrote: post #201
oceanplexian wrote: post #149

Earlier quoted context omitted.

Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise. Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.

"Content you're allowed to scrape from the internet" is MUCH smaller than what LLMs have actually scraped, but they don't care about copyright. > The fears that we have “used all the data” are incredibly unfounded. The problem isn't whether we used all the real data or not, the problem is that it becomes increasingly difficult to distinguish real data from previous LLM outputs.

> "Content you're allowed to scrape from the internet" is MUCH smaller than what LLMs have actually scraped, but they don't care about copyright.

I don't know about that. If you scraped the same data and ran a search engine I think people would generally say you're fine. The copyright issue isn't the scraping step.

Re: Anyone got a contact at OpenAI. They have a spider problem

#252

Post by jpambrun »

lobsterthief wrote: post #170
Jordan-117 wrote: post #129

Earlier quoted context omitted.

It's not just a problem for training, but the end user, too. There are so many times that I've tried to ask a question or request a summary for a long article only to be told it can't read it itself, so you have to copy-paste the text into the chat. Given the non-binding nature of robots.txt and the way they seem comfortable with vacuuming up public data in other contexts, I'm surprised they allow it to be such an ob…

That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead. It’s functioning exactly as designed.

It's a stretch to expect a human initiated action to abide by robot.txt.

Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..

Re: Anyone got a contact at OpenAI. They have a spider problem

#253

Post by colonelspace »

jpambrun wrote: post #252
lobsterthief wrote: post #170

Earlier quoted context omitted.

That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead. It’s functioning exactly as designed.

It's a stretch to expect a human initiated action to abide by robot.txt. Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..

At bottom, all robot actions are human initiated.

Re: Anyone got a contact at OpenAI. They have a spider problem

#254

Post by brookst »

mindcrime wrote: post #245
singingfish wrote: post #233

Earlier quoted context omitted.

I'm really cross that the word "hallucination" has taken off to describe this as it's clearly in incorrect word. The correct word to describe it is "confabulation", which is clinically more accurate and a much clearer descriptor of what's actually going on. https://en.wikipedia.org/wiki/Confabulation

I proposed[1] the portmanteau "hallucofabulation" as a compromise, but it hasn't caught on yet. I'm totally shocked and dismayed by this, of course. [1]: https://news.ycombinator.com/item?id=36977935

The re-use of the "c" as a soft c in "hallucinate" and then a hard c in confabulate is confusing, and probably affecting the uptake of your neologism.

Re: Anyone got a contact at OpenAI. They have a spider problem

#255

Post by Zambyte »

lobsterthief wrote: post #170
Jordan-117 wrote: post #129

Earlier quoted context omitted.

It's not just a problem for training, but the end user, too. There are so many times that I've tried to ask a question or request a summary for a long article only to be told it can't read it itself, so you have to copy-paste the text into the chat. Given the non-binding nature of robots.txt and the way they seem comfortable with vacuuming up public data in other contexts, I'm surprised they allow it to be such an ob…

That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead. It’s functioning exactly as designed.

I would say robots.txt is meant to filter access for interactions initiated by an automated process (ie automatic crawling). Since the interaction to request a site with a language model is manual (a human request) it doesn't make sense to me that it is used to block that request.

If you want to block information you provide from going through ClosedAI servers, block their IPs instead of using robots.txt.

Re: Anyone got a contact at OpenAI. They have a spider problem

#256

Post by SrslyJosh »

ezekg wrote: post #71

Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to so…

Everyone saying "ouroboros": The phrase you're looking for is "human centipede". =)

Re: Anyone got a contact at OpenAI. They have a spider problem

#257

Post by _pdp_ »

In the network security world, this is known as a tarpit. You can delay an attack, scan or any other type of automation by sending data either too slowly or in such a way as to cause infinite recursion. The result is wasted time and energy for the attacker and potentially a chance for us to ramp up the defences.

Re: Anyone got a contact at OpenAI. They have a spider problem

#258

Post by Zambyte »

Tubbe wrote: post #246
cactusplant7374 wrote: post #174

Earlier quoted context omitted.

The CTO isn't even aware of where the data is coming from (allegedly).

(admittedly)

The (allegedly) implies they do know, but to avoid possible litigation they feign ignorance. The CTO of ClosedAI is probably not a complete idiot.

Re: Anyone got a contact at OpenAI. They have a spider problem

#259

Post by 1317 »

He's not done his robots.txt properly, he's commented out the bit that actually disallows it

  # silly bing
  #User-agent: Amazonbot          
  #Disallow: /

  # buzz off
  #User-agent: GPTBot
  #Disallow: /

  # Don't Allow everyone
  User-agent: *
  Disallow: /archive

  # slow down, dudes
  #Crawl-delay: 60

Re: Anyone got a contact at OpenAI. They have a spider problem

#260

Post by dizhn »

TeMPOraL wrote: post #193
jdashg wrote: post #139

Earlier quoted context omitted.

Nowadays they are instead learning to say "please join our Discord for support"!

Much like one of the first phrases spoken by babies today is "like and subscribe".

For the actual useful information visit my Patreon.
Post reply on HN