Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

281–290 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#281

Post by Dr_Birdbrain »

bevekspldnw wrote: post #181
Eisenstein wrote: post #152

Earlier quoted context omitted.

> Facebook alone probably has more data than the entire dataset GPT4 was trained on and it’s all behind closed doors. Meta is happily training their own models with this data, so it isn't going to waste.

Not Llama, they’ve been really clear about that. Especially with DMA cross-joining provisions and various privacy requirements it’s really hard for them, same for Google. However, Microsoft has been flying under the radar. If they gave all Hotmail and O365 data to OpenAI I’d not be surprised in the slightest.

I bet they are training their internal models on the data. Bet the real reason they are not training open source models on that data is because of fears of knowledge distillation, somebody else could distill LLaMa into other models. Once the data is in one AI, it can be in any AIs. This problem is of course exacerbated by open source models, but even closed models are not immune, as the Alpaca paper showed.

Re: Anyone got a contact at OpenAI. They have a spider problem

#282

Post by Karellen »

imnotreallynew wrote: post #21

Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…

> Isn’t the legality of web scraping still..disputed?

Are you suggesting it might be illegal to... write a program that connects to a web server and asks for a specific page, and then parses that page to see which resources it wants and which other pages it links to, and treats those links in some special fashion, differently from the text content of the page?

Especially given that a web server can be configured to respond to any request with a "403 Forbidden" response, if the server determines for any reason whatsoever that it does not want to give the client the page it requested?

Re: Anyone got a contact at OpenAI. They have a spider problem

#283

Post by bityard »

a_c wrote: post #72
queuebert wrote: post #60

Earlier quoted context omitted.

Did we just figure out a DoS attack for AGI training? How large can a robots.txt file be?

What about making it slow? One byte at a time for example while keeping the connection open

That would make it a tarpit, a very old technique to combat scrapers/scanners

Re: Anyone got a contact at OpenAI. They have a spider problem

#284

Post by roughly »

microtherion wrote: post #195
gunapologist99 wrote: post #110

Earlier quoted context omitted.

This must be why self-driving cars always ignore the speed limit. ;)

More directly, e.g. Tesla boasts of training their FSD on data captured from their customer's unassisted driving. So it's hardly surprising that it imitates a lot of humans' bad habits, e.g. rolling past stop lines.

Jesus, that’s one of those ideas that looks good to an engineer but is why you really need to hire someone with a social sciences background (sociology, anthropology, psychology, literally anyone who’s work includes humans), and probably should hire two, so the second one can tell you why the first died of an aneurism after you explained your idea.

Re: Anyone got a contact at OpenAI. They have a spider problem

#285

Post by bityard »

_pdp_ wrote: post #257

In the network security world, this is known as a tarpit. You can delay an attack, scan or any other type of automation by sending data either too slowly or in such a way as to cause infinite recursion. The result is wasted time and energy for the attacker and potentially a chance for us to ramp up the defences.

From the content of the email, I get the impression that it's just a honeypot. Also I'm not seeing any delays in the content being returned.

A tarpit is different because it's designed to slow down scanning/scraping and deliberately waste an adversary's resources. There are several techniques but most involve throttling the response (or rate of responses) exponentially.

Re: Anyone got a contact at OpenAI. They have a spider problem

#286

Post by Calzifer »

starttoaster wrote: post #155
throw_a_grenade wrote: post #135

Earlier quoted context omitted.

You just set limits on everything (time, buffers, ...), which is easier said than done. You need to really understand your libraries and all the layers down to the OS, because its enough to have one abstraction that doesn't support setting limits and it's an invitation for (counter-)abuse.

Doesn't seem like it should be all that complex to me assuming the crawler is written in a common programming language. It's a pretty common coding pattern for functions that make HTTP requests to set a timeout for requests made by your HTTP client. I believe the stdlib HTTP library in the language I usually write in actually sets a default timeout if I forget to set one.

Those are usually connection and no-data timeouts. A total time limit is in my experience less common.

Re: Anyone got a contact at OpenAI. They have a spider problem

#287

Post by mindcrime »

brookst wrote: post #254
mindcrime wrote: post #245

Earlier quoted context omitted.

I proposed[1] the portmanteau "hallucofabulation" as a compromise, but it hasn't caught on yet. I'm totally shocked and dismayed by this, of course. [1]: https://news.ycombinator.com/item?id=36977935

The re-use of the "c" as a soft c in "hallucinate" and then a hard c in confabulate is confusing, and probably affecting the uptake of your neologism.

Maybe if I added a hyphen? "halluco-fabulation"?

Re: Anyone got a contact at OpenAI. They have a spider problem

#288

Post by dkasper »

singingfish wrote: post #276
dkasper wrote: post #263

Earlier quoted context omitted.

That’s a great word for some types of hallucinations. But some things that are called hallucinations may not be memory errors.

Please can you give an example of what might not be a memory error. Not that I think "memory error" is the right phrase either.

I was thinking along the lines of answering with correct information but not following the prompts. Maybe this could be considered confabulation also.

Re: Anyone got a contact at OpenAI. They have a spider problem

#289

Post by symlinkk »

Isn’t it funny that all the “worthless” content out there on the internet is actually changing the world. Like how 4chan was mocked as being a cesspit for losers, but now everyone knows memes like Pepe the frog and Wojack from there. And like now this very comment and the billions of other comments on here, Reddit, Twitter, etc that are regarded as a “waste of time” are being used to train multi billion dollar companies to build the most powerful AI the world has ever seen. For free.

The moral of the story here is if you know something valuable, don’t share it online, because then everyone knows it.

Re: Anyone got a contact at OpenAI. They have a spider problem

#290

Post by rstuart4133 »

Dylan16807 wrote: post #274
tsunamifury wrote: post #240

Earlier quoted context omitted.

Hackernews really instant able to understand dark jokes are they.

You didn't really do the joke right. The second person is supposed to be doing the thing requested, but in a way the first person doesn't like. In this case, that would be AI companies contributing to a "free and open internet", but doing it """wrong""". But they're not contributing at all. The problem isn't that "free and open internet" is getting monkeys-pawed, it's all this closed proprietary stuff.

> You didn't really do the joke right.

It summed up many comments and attitudes I see here on LLM's in under 30 words. In fact it's so clever it's one of the few posts here I'm certain wasn't produced by an LLM.

Post reply on HN