Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

41–50 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#41

Post by cosmojg »

euparkeria wrote: post #27

What is a spider in this context?

> A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

See: https://en.wikipedia.org/wiki/Web_crawler

Re: Anyone got a contact at OpenAI. They have a spider problem

#43

Post by relaxing »

tomfreemax wrote: post #16
McBun wrote: post #6

[flagged]

I have to say I don't really get the website either. If the author is against scraper why not serve massive dummy content that it bloats their storage? Why all this linking? Maybe it's used to build (fake) page rank credibility and sometimes a link to one of the content farm pages is referenced on other pages, so these get boosted then?

Presumably he would be paying for egress of those massive files?

Re: Anyone got a contact at OpenAI. They have a spider problem

#44

Post by Octokiddie »

gtirloni wrote: post #38
Octokiddie wrote: post #26

I'm more interested in what that content farm is for. It looks pointless, but I suspect there's a bizarre economic incentive. There are affiliate links, but how much could that possibly bring in?

It'd say it's more like a honeypot for bots. So pretty similar objectives.

So it served its purpose by trapping the OpenAI spider? If so, why post that message? As a flex?

Re: Anyone got a contact at OpenAI. They have a spider problem

#45

Post by dimask »

ganzuul wrote: post #42
alberth wrote: post #19

Isn't the entire point of these type of websites to waste spider time/resources. Why do they want to not do that for OpenAI?

Might one day come looking for who lobotomized it?

And? What are they gonna do about it (apart from making such a person/website momentarily famous).

Re: Anyone got a contact at OpenAI. They have a spider problem

#46

Post by dspillett »

gtirloni wrote: post #38
Octokiddie wrote: post #26

I'm more interested in what that content farm is for. It looks pointless, but I suspect there's a bizarre economic incentive. There are affiliate links, but how much could that possibly bring in?

It'd say it's more like a honeypot for bots. So pretty similar objectives.

So, it has worked…

Re: Anyone got a contact at OpenAI. They have a spider problem

#47

Post by madkangas »

Octokiddie wrote: post #26

I'm more interested in what that content farm is for. It looks pointless, but I suspect there's a bizarre economic incentive. There are affiliate links, but how much could that possibly bring in?

I recognize the name John Levine at iecc.com, "Invincible Electric Calculator Company," from web 1.0 era. He was the moderator of the Usenet comp.compilers newsgroup and wrote the first C compiler for the IBM PC RT

https://compilers.iecc.com/

Re: Anyone got a contact at OpenAI. They have a spider problem

#48

Post by RobotToaster »

ganzuul wrote: post #42
alberth wrote: post #19

Isn't the entire point of these type of websites to waste spider time/resources. Why do they want to not do that for OpenAI?

Might one day come looking for who lobotomized it?

Do androids dream of electric farms?

Re: Anyone got a contact at OpenAI. They have a spider problem

#49

Post by pvg »

YeGoblynQueenne wrote: post #32
pvg wrote: post #22

Earlier quoted context omitted.

What am I looking at? I'd say go ahead and inject it with digestive enzymes and then report findings.

No no. First tightly wrap it in silk. Then digestive enzymes. Good hygiene, eh?

Largely a myth spread by Big Silk!

Re: Anyone got a contact at OpenAI. They have a spider problem

#50

Post by Takennickname »

Octokiddie wrote: post #44
gtirloni wrote: post #38

Earlier quoted context omitted.

It'd say it's more like a honeypot for bots. So pretty similar objectives.

So it served its purpose by trapping the OpenAI spider? If so, why post that message? As a flex?

It's a honeypot. He's telling people openai doesn't respect robots.txt and just scrapes whatever the hell it wants.
Post reply on HN