Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

161–170 of 279 posts

Re: We can't have nice things because of AI scrapers

#161

Post by philipwhiuk »

Olshansky wrote: post #83
bediger4000 wrote: post #49

Earlier quoted context omitted.

At this point, it's pretty clear that the AI scrapers won't be limited by any voluntary restrictions. Bytedance never seemed to live with robots.txt limitations, and I think at least some of the others didn't either. I can't see this working.

The thesis/approach is: - Humans tip humans as a lottery ticket for an experience (meet the creator) or sweepstakes (free stuff) - Agents tip humans because they know they'll need original online content in the long-term to keep improving. For the latter, frontier labs will need to fund their training/inference agents with a tipping jar. There's no guarantee, but I can see it happening given where things are movin.

> Agents tip humans because they know they'll need original online content in the long-term to keep improving.

Why would an agent have any long term incentive. It's trained to 'do what it's told', not to predict the consequences of it's actions.

Re: We can't have nice things because of AI scrapers

#162

Post by philipwhiuk »

TeMPOraL wrote: post #131
VonGallifrey wrote: post #118

Earlier quoted context omitted.

> turns out plenty of people say they write for others to read LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.

LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access . The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.

And rarely cite their sources, thus affording the author not so much a crumb of benefit in kind.

Re: We can't have nice things because of AI scrapers

#163

Post by chlorion »

tommek4077 wrote: post #108
tclancy wrote: post #61

Earlier quoted context omitted.

Yes yes, definitely people don’t know what they’re doing and not that they’re operating on a scale or problem you are not. Metabrainz cannot cache all of these links as most of them are hardly ever hit. Try to assume good intent.

But serving HTML is unbelievably cheap, isn't it?

It adds up very quickly.

Re: We can't have nice things because of AI scrapers

#164

Post by DrewADesign »

godelski wrote: post #144
DrewADesign wrote: post #135

Earlier quoted context omitted.

They should advertise that. I pretty much reflexively avoid any mention of AI in interfaces because they usually mean "we're sending this all to openthropigoogosoft so I hope you don't have any secrets."

> They should advertise that They did Previews can optionally include AI-generated key points, which are processed on your device to protect your privacy. https://www.firefox.com/en-US/firefox/142.0/releasenotes/ I'll also add that if you go to the Labs page (in settings) you can enable another local model to semantically search your history

Ok, they should advertise it more .

Re: We can't have nice things because of AI scrapers

#165

Post by ninkendo »

lysace wrote: post #136
juliangmp wrote: post #133

Earlier quoted context omitted.

"Why don't you just clone the repo?" Yes. Why dont you? If you're gonna grab a repo to make a code theft machine then at least dont ddos the servers while you're at it.

[flagged]

Why don’t you take a moment to explain to the class why you think web crawling means you can’t cache anything?

It seems to me that the very first thing I’d try to solve if I were writing a tool for an LLM to search the web, would be caching.

An LLM should have to go through a proxy to fetch any URL. That proxy should be caching results. The cache should be stored on the LLM’s company’s servers. It should not be independently hitting the same endpoint repeatedly any time it wants to fetch the same URL for its users.

Is it expensive to cache everything the LLM fetches? You betcha. Can they afford to spend of the billions they have for capex to buy some fucking hard drives? Absolutely. If archive.org can do it via funding from donations, a trillion dollar AI company should have no problem.

Re: We can't have nice things because of AI scrapers

#166

Post by everybodyknows »

bodantogat wrote: post #19

I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.

> residential IP pools

So, is this a new profit center for sleazeball household ISPs?

Re: We can't have nice things because of AI scrapers

#169

Post by Forgeties79 »

NegativeK wrote: post #128
lysace wrote: post #113

Earlier quoted context omitted.

[flagged]

I've downvoted you for being incredibly aggressive in your responses. I'm not sure why you're ad homineming the parent commenter, but it's not helping the discussion.

I don’t even really get what they are saying. I am also saying that they are hostile, and with all of their money they can afford to not be hostile. So I feel like we agree?

Re: We can't have nice things because of AI scrapers

#170

Post by 8note »

nostrademons wrote: post #17
tensegrist wrote: post #8

the more time passes the more i'm convinced that the solution is to—somehow—force everyone to have to go through something like common crawl i don't want people's servers to be pegged at 100% because a stupid dfs scraper is exhaustively traversing their search facets, but i also want the web to remain scrapable by ordinary people, or rather go back to how readily scrapable it used to be before the invention of cloudf…

That's sorta what MetaBrainz did - they offer their whole DB as a single tarball dump, much like what Wikipedia does. I downloaded it in the order of an hour; if I need a MusicBrainz lookup, I just do a local query. For this strategy to work, people need to actually use the DB dumps instead of just defaulting to scraping. Unfortunately scraping is trivially easy, particularly now that AI code assistants can write a w…

the obvious thing would be to take down their website and only have the DB dump.

if thats the useful thing, it doesnt need the wrapper

Post reply on HN