Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTBot – OpenAI’s Web Crawler

platform.openai.com

Post reply on HN
Print view View original

111–120 of 327 posts

Re: GPTBot – OpenAI’s Web Crawler

#111

Post by gfedtbyby »

nomilk wrote: post #17
wg0 wrote: post #5

If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving

This doesn't appear consistent with other visitors to your website. If a cafe owner uses info on your site to improve their baking, should they also be required to share their revenue with you?

If the cafe is a multi-billion corporation that can only exist because it can leech of content created by millions of other people without providing anything at all to them in return (and I'm not necessarily talking about financial compensation) then yeah.. maybe you should.

Re: GPTBot – OpenAI’s Web Crawler

#112

Post by golol »

ketzo wrote: post #16
wg0 wrote: post #5

If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving

Counterpoint (not just to be annoying — I think you pose a very interesting unanswered question): If I read your hobby website about photography and use it to take 1% better pictures, do I owe you 1% of what my clients pay me? I think that probably most people would say no, assuming you could even determine that 1% in a way that both parties agreed was fair. I think generally, we have an understanding that some stuff…

It should be treated as learing. If it truly stores and reproduces a photo (to some high accuracy), then there are already laws in place that handle this. Your client using the output may infringe on the photographer's rights, which may fall back on you depending on your contract.

If I watch a youtube video my browser is also in a way scraping youtube and storing a (temporary) copy of the video. Does it make sense to protect the protect the owner's right's at this point? Absolutely not. Instead we wait to see if I share that downloaded video or content from it again, or somehow reuse it in my own products. Only then does the law step in.

Re: GPTBot – OpenAI’s Web Crawler

#113

Post by p-e-w »

croes wrote: post #40
p-e-w wrote: post #32

Earlier quoted context omitted.

So there's an infinite pyramid of "who learned what from who", and payment flows upwards along the hierarchy, all the way back to people who are long dead, and then down to their descendants who presumably inherited their "knowledge rights"? You can't be serious. Thank god our world doesn't work like that.

It's not about who learned what from whom, it's about superstar economy. If you serve all customers and leave nothing for the rest, it will be a problem. What do you think why writers and actors have included AI in the reasons of their strike?

> What do you think why writers and actors have included AI in the reasons of their strike?

Because they are about to become obsolete, and they believe that screaming as loudly as they can is going to stop that.

Their chances of success are roughly the same as if they were protesting against the law of gravity.

Re: GPTBot – OpenAI’s Web Crawler

#114

Post by keymon-o »

awb wrote: post #58
keymon-o wrote: post #48

Earlier quoted context omitted.

A human reads a website, watches/clicks on ad, buys merch, subscribes to website, sets a bookmark to a website, shares an article, invites others etc. What will be the point of sharing knowledge or content if it will no longer be associated to an individual or organization?

That sounds like an argument against any bot visiting a monetized website. Some people publish content freely on the Internet as a form of note taking, publicity, public discourse or for the betterment of like minded individuals, akin to why we’re here commenting on HN. I guess I assume public content defaults into this “for the benefit of the world” category, where it’s up to the publisher to gate content as desired…

No, that's an argument against any bot visiting any website for the purpose of repackaging and redistributing information regardless of motivation website was concieved for.

Public content is still mostly published with a reference to a certain or anonymous individual/organization and gains visibility based on a value and an effort to be seen. Individual/organization is still motivated by visibility, popularity, acceptance and approval of that content.

We can summarize that people are motivated by a reaction. What do you think will happen when you remove or decrease reaction to knowledge/content providers?

Re: GPTBot – OpenAI’s Web Crawler

#115

Post by gfedtbyby »

visarga wrote: post #18
wg0 wrote: post #5

If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving

> shouldn't I get some free credits to use that model or proportionate share in the revenue stream But you do get paid in kind - you "gave" information for the AI to train on, the aI gives you information back, contextualised to your needs. Sometimes those 1000 tokens are worth much more than $0.06 You still need to be able to pay for inference costs, it's crowded and expensive on GPUs nowadays.

Since it "gives" the same information to everyone there aren't really that many incentives for you to allow LLM to use your content. "tragedy of the commons" and all that stuff...

Re: GPTBot – OpenAI’s Web Crawler

#116

Post by golol »

supriyo-biswas wrote: post #20
losteric wrote: post #14

Earlier quoted context omitted.

Why? I wouldn't pay you for marginally improving my baking skills either. It is an interesting question. I would have no qualms paying for a textbook or university course for curated learning (worth noting OpenAI has paid datasets too), but paying for (or being paid for) relatively diffuse and low quality content through hobby blogs seems at odds with my expectations as an individual, and as a society we were never (…

Because perfect information transfer isn’t usually possible by a human reading a book or website, whereas computer systems can usually do that. If humans could perfectly remember information, I’m sure copyright would be very different.

But a model learning from data and reproducing it in some fashion is absolutely not perfect information transfer.

Re: GPTBot – OpenAI’s Web Crawler

#117

Post by golol »

croes wrote: post #27
p-e-w wrote: post #21

Earlier quoted context omitted.

If I learn something from your StackOverflow answers, do you expect me to share a percentage of my future salary with you?

Can you share your knowledge with millions at once? If so, then pay.

> Can you share your knowledge with millions at once?

Yes, I might run a course or something. You are still not entitled to pay.

Re: GPTBot – OpenAI’s Web Crawler

#118

Post by gfedtbyby »

drekipus wrote: post #9
wg0 wrote: post #5

If you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving

quick back-of-envelope/googling: openai is worth $29,000,000.00 you contributed 0.00000000001 punches numbers in calculator thus the value of your free credits is 0.001 cents. minus any accounting fees.

Much of that is Azure credits and not real money.

Re: GPTBot – OpenAI’s Web Crawler

#119

Post by gmerc »

Zaheer wrote: post #4

Nice of them to respect crawling after they've already trained their model. Presumably these headers don't affect any pages they've already crawled to train GPT(?)

It’s so now they can lobby for anti scraping regulation and hamper any possible catch-up.

Re: GPTBot – OpenAI’s Web Crawler

#120

Post by uallo »

colechristensen wrote: post #94
uallo wrote: post #39

Yet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death. Luckily, I already block the IP address they're using as it has been used for (other?) malicious bots before. [1] In my case, it relies on third-party APIs that are heavily rate limited. Any bot ignoring rate limitation measures will effectively (D)DOS my serv…

There are plenty of ways you can (and should) rate limit requests on your end. It is a pretty basic security and reliability practice. Also if you're dealing with an actual malicious adversary real or automated rate limiting can be more effective than blocking. (logic to detect and overcome an even very significant rate limit is much more complex than to detect dropping, ignoring, or 4xx 5xx response blocking methods…

Sure. I already use several rate limitation measures, return fake data for repeating offenders, and also outright block some others. It is still laughable that a somewhat "reputable" bot does not even know about basic HTTP headers.
Post reply on HN