Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

91–100 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#91

Post by abracadaniel »

aidenn0 wrote: post #86
btown wrote: post #30

This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting , wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAx…

I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…

That’s a good observation. If LLMs had taken off 15 years ago, maybe they would answer every question with “this has already been asked before. Please use the search function”

Re: Anyone got a contact at OpenAI. They have a spider problem

#92

Post by everforward »

gardenhedge wrote: post #70
danpalmer wrote: post #63

Earlier quoted context omitted.

A "honeypot" is a system designed to trap unsuspecting entrants. In this case, the website is designed to be found by web crawlers and to then trap them in never-ending linked sites that are all pointless. Other honeypots include things like servers with default passwords designed to be found by hackers so as to find the hackers.

What does trap mean here? I presumed crawlers had multiple (thousands of or more) instances. One being 'trapped' on this web farm won't have any impact

I would presume the crawlers have a queue-based architectures with thousands of workers. It’s an amplification attack.

When a worker gets a webpage for the honeypot, it crawls it, scrapes it, and finds X links on the page where X is greater than 1. Those links get put on the crawler queue. Because there’s more than 1 link per page, each worker on the honeypot will add more links to the queue than it removed.

Other sites will eventually leave the queue, because they have a finite number of pages so the crawlers eventually have nothing new to queue.

Not on the honeypot. It has a virtually infinite number of pages. Scraping a page will almost deterministically increase the size of the queue (1 page removed, a dozen added per scrape). Because other sites eventually leave the queue, the queue eventually becomes just the honeypot.

OpenAI is big enough this probably wasn’t their entire queue, but I wouldn’t be surprised if it was a whole digit percentage. The author said 1.8M requests; I don’t know the duration, but that’s equivalent to 20 QPS for an entire day. Not a crazy amount, but not insignificant. It’s within the QPS Googlebot would send to a fairly large site like LinkedIn.

Re: Anyone got a contact at OpenAI. They have a spider problem

#93

Post by ImPostingOnHN »

relaxing wrote: post #43
tomfreemax wrote: post #16

Earlier quoted context omitted.

I have to say I don't really get the website either. If the author is against scraper why not serve massive dummy content that it bloats their storage? Why all this linking? Maybe it's used to build (fake) page rank credibility and sometimes a link to one of the content farm pages is referenced on other pages, so these get boosted then?

Presumably he would be paying for egress of those massive files?

So render it clientside and hope the crawler understands javascript?

Maybe run your own training in javascript, too, and use OpenAI's crawlers' compute for it.

Re: Anyone got a contact at OpenAI. They have a spider problem

#94

Post by swyx »

flutas wrote: post #65
GaggiX wrote: post #57

Earlier quoted context omitted.

It seems to respect it as the majority of the requests are for the robots.txt.

He says 3 million, and 1.8 million are for robots.txt So 1.2 million non robots.txt requests, when his robots.txt file is configured as follows # buzz off User-agent: GPTBot Disallow: / Theoretically if they were actually respecting robots.txt they wouldn't crawl any pages on the site. Which would also mean they wouldn't be following any links... aka not finding the N subdomains.

for the 1.2 million are there other links he's not telling us about?

Re: Anyone got a contact at OpenAI. They have a spider problem

#95

Post by exe34 »

abracadaniel wrote: post #91
aidenn0 wrote: post #86

Earlier quoted context omitted.

I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…

That’s a good observation. If LLMs had taken off 15 years ago, maybe they would answer every question with “this has already been asked before. Please use the search function”

Marked as duplicate.

Re: Anyone got a contact at OpenAI. They have a spider problem

#96

Post by brushfoot »

bena wrote: post #66
brushfoot wrote: post #29

Earlier quoted context omitted.

Web scraping the public Internet is legal, at least in the U.S. hiQ's public scraping of LinkedIn was ruled to be within their rights and not a violation of the CFAA. I imagine that's why LinkedIn has almost everything behind an auth wall now. Scraping auth-walled data is different. When you sign up, you have to check "I agree to the terms," and the terms generally say, "You can't scrape us." So, you can't just make…

hiQ was found to be in violation of the User Agreement in the end. Basically, in the end, it was essentially a breach of contract.

Exactly, that was my point.

hiQ's public scraping was found to be legal. It was the logged-in scraping that was the problem.

The logged-in scraping was a breach of contract, as you said.

The former is fine; the latter is not.

What OpenAI is doing here is the former, which companies are perfectly within their rights to do.

Re: Anyone got a contact at OpenAI. They have a spider problem

#97

Post by otherme123 »

flutas wrote: post #65
GaggiX wrote: post #57

Earlier quoted context omitted.

It seems to respect it as the majority of the requests are for the robots.txt.

He says 3 million, and 1.8 million are for robots.txt So 1.2 million non robots.txt requests, when his robots.txt file is configured as follows # buzz off User-agent: GPTBot Disallow: / Theoretically if they were actually respecting robots.txt they wouldn't crawl any pages on the site. Which would also mean they wouldn't be following any links... aka not finding the N subdomains.

A lot of crawlers, if not all, have a policy like "if you disallow our robot, it might take a day or two before it notices". They surely follow the path "check if we have robots.txt that allows us to scan this site, if we don't get and store robots.txt, scan at least the root of the site and its links". There won't be a second scan, and they consider that they are respecting robots.txt. Kind of "better ask for forgiveness than for permission".

Re: Anyone got a contact at OpenAI. They have a spider problem

#99

Post by pants2 »

sarah_eu wrote: post #80
ezekg wrote: post #71

Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to so…

Well it will be multimodal, training and inferring on feeds of distributed sensing networks; radio, optical, acoustic, accelerometer, vibration, anything that's in your phone and much besides. I think the time of the text-only transformer has already passed.

OpenAI will just litter microphones around public spaces to record conversations and train on them.

Re: Anyone got a contact at OpenAI. They have a spider problem

#100

Post by fsckboy »

darkwater wrote: post #68
cwillu wrote: post #55

Earlier quoted context omitted.

Except the first thing openai does is read robots.txt. However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.

And they do have (the same) robots.txt on every domain, tailored for GPTbot, i.e. https://petra-cody-carlene.web.sp.am/robots.txt So, GPTBot is not following robots.txt, apparently.

humans don't read/respect robots.txt, so in order to pass the Turing test, ai's need to mimic human behavior.
Post reply on HN