Anyone got a contact at OpenAI. They have a spider problem • Hacker News Board

Re: Anyone got a contact at OpenAI. They have a spider problem

#11

Post by mapmeld » Thu, Apr 11, 2024, 2:10 PM UTC

egberts1 wrote: post #5

There is always IP filtering, DNS blocking, and HTTP agent screening. Just sayin'.

Based on the page footer ("IECC ChurnWare") I believe this is a site by design to waste time for web crawlers and tools which try to get root access on every domain. The robots.txt looks like this: https://ulysses-antoine-kurtis.web.sp.am/robots.txt

I don't see how this does much to keep bad actors away from other domains, but I can see why they don't want to give up the game for OpenAI to stop crawling.

Re: Anyone got a contact at OpenAI. They have a spider problem

#12

Post by spacecadet » Thu, Apr 11, 2024, 2:10 PM UTC

McBun wrote: post #6

[flagged]

Yeah great honeypot

Re: Anyone got a contact at OpenAI. They have a spider problem

#13

Post by [deleted] » Thu, Apr 11, 2024, 2:11 PM UTC

McBun wrote: post #6

[flagged]

[deleted]

Re: Anyone got a contact at OpenAI. They have a spider problem

#14

Post by aendruk » Thu, Apr 11, 2024, 2:13 PM UTC

mypastself wrote: post #9

Frankly, I didn’t get the purpose of the website at first either. I guess I have an arachnid intellect.

Arachnid here… What am I looking at? The intent is to waste the resources of crawlers by just making the web larger?

Re: Anyone got a contact at OpenAI. They have a spider problem

#15

Post by frizlab » Thu, Apr 11, 2024, 2:14 PM UTC

I’d let them do their thing, why not?! They want the internet? This is the real internet. It looks like he doesn’t really care that much that they’re retrieving millions of pages, so let them do their thing…

Re: Anyone got a contact at OpenAI. They have a spider problem

#16

Post by tomfreemax » Thu, Apr 11, 2024, 2:16 PM UTC

McBun wrote: post #6

[flagged]

I have to say I don't really get the website either. If the author is against scraper why not serve massive dummy content that it bloats their storage? Why all this linking? Maybe it's used to build (fake) page rank credibility and sometimes a link to one of the content farm pages is referenced on other pages, so these get boosted then?

Re: Anyone got a contact at OpenAI. They have a spider problem

#17

Post by [deleted] » Thu, Apr 11, 2024, 2:18 PM UTC

mypastself wrote: post #9

Frankly, I didn’t get the purpose of the website at first either. I guess I have an arachnid intellect.

[deleted]

Re: Anyone got a contact at OpenAI. They have a spider problem

#18

Post by quadrature » Thu, Apr 11, 2024, 2:19 PM UTC

aendruk wrote: post #14

mypastself wrote: post #9

Frankly, I didn’t get the purpose of the website at first either. I guess I have an arachnid intellect.

Arachnid here… What am I looking at? The intent is to waste the resources of crawlers by just making the web larger?

It seems like that, but they're also concerned about the crawlers that they catch in this web. So it seems like they're trying to help make crawlers better ?, or just generally curious about what systems are crawling around.

Re: Anyone got a contact at OpenAI. They have a spider problem

#19

Post by alberth » Thu, Apr 11, 2024, 2:19 PM UTC

Isn't the entire point of these type of websites to waste spider time/resources.

Why do they want to not do that for OpenAI?

Re: Anyone got a contact at OpenAI. They have a spider problem

#20

Post by unnouinceput » Thu, Apr 11, 2024, 2:20 PM UTC

fl7305 wrote: post #10

egberts1 wrote: post #5

There is always IP filtering, DNS blocking, and HTTP agent screening. Just sayin'.

I think he's saying that it's not a problem for him, but for OpenAI?

Yup, that's my impression as well. He's just nice to let OpenAI they have a problem. Usually this should be rewarded with a nice "hey, u guys have a bug" bounty because not long time ago some VP from OpenAI was lamenting that training their AI is, and it's his direct quote, "eye watering" cost (the order was millions of $$ per second).