Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

21–30 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#21

Post by imnotreallynew »

Isn’t the legality of web scraping still..disputed?

There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible.

It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats.

Was there some law that was passed that makes all web scraping legal or something?

Re: Anyone got a contact at OpenAI. They have a spider problem

#22

Post by pvg »

aendruk wrote: post #14
mypastself wrote: post #9

Frankly, I didn’t get the purpose of the website at first either. I guess I have an arachnid intellect.

Arachnid here… What am I looking at? The intent is to waste the resources of crawlers by just making the web larger?

What am I looking at?

I'd say go ahead and inject it with digestive enzymes and then report findings.

Re: Anyone got a contact at OpenAI. They have a spider problem

#23

Post by [deleted] »

imnotreallynew wrote: post #21

Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…

[deleted]

Re: Anyone got a contact at OpenAI. They have a spider problem

#24

Post by foobarian »

imnotreallynew wrote: post #21

Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…

Why would it not be legal? Was there a law passed that makes it illegal?

Re: Anyone got a contact at OpenAI. They have a spider problem

#28

Post by apocalyptic0n3 »

unsupp0rted wrote: post #7
egberts1 wrote: post #5

There is always IP filtering, DNS blocking, and HTTP agent screening. Just sayin'.

> Before someone tells me to fix my robots.txt, this is a content farm so rather than being one web site with 6,859,000,000 pages, it is 6,859,000,000 web sites each with one page.

The reason that bit is relevant is that robots.txt is only applicable to the current domain. Because each "page" is a different subdomain, the crawler needs to fetch the robots.txt for every single page request.

What the poster was suggesting is blocking them at a higher level - e.g. a user-agent block in an .htaccess or an IP block in iptables or similar. That would be a one-stop fix. It would also defeat the purpose of the website, however, which is to waste the time of crawlers

Re: Anyone got a contact at OpenAI. They have a spider problem

#29

Post by brushfoot »

imnotreallynew wrote: post #21

Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…

Web scraping the public Internet is legal, at least in the U.S.

hiQ's public scraping of LinkedIn was ruled to be within their rights and not a violation of the CFAA. I imagine that's why LinkedIn has almost everything behind an auth wall now.

Scraping auth-walled data is different. When you sign up, you have to check "I agree to the terms," and the terms generally say, "You can't scrape us." So, you can't just make a million bot accounts that take an app's data (legally, anyway). Those EULAs are generally legally enforceable in the U.S.

Some sites have terms at the bottom that prohibit scraping—but my understanding is that those aren't generally enforceable if the user doesn't have to take any action to accept or acknowledge them.

Re: Anyone got a contact at OpenAI. They have a spider problem

#30

Post by btown »

This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting, wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own.

https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid...

https://www.lesswrong.com/posts/LAxAmooK4uDfWmbep/anomalous-...

Vocabulary isn't infinite, and GPT-3 reportedly had only 50,257 distinct tokens in its vocabulary. It does make me wonder - it's certainly not a linear relationship, but given the number of inferences run every day on GPT-3 while it was the flagship model, the incremental electricity cost of these Redditors' niche hobby, vs. having allocated those slots in the vocabulary to actually common substrings in real-world text and thus reducing average input token count, might have been measurable.

It would be hilarious if the subtitle on OP's site, "IECC ChurnWare 0.3," became a token in GPT-5 :)

Post reply on HN