Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miasma: A tool to trap AI web scrapers in an endless poison pit

github.com

Post reply on HN
Print view View original

161–170 of 276 posts

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#161

Post by Bender »

spiderfarmer wrote: post #32
tasuki wrote: post #28

> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

Depends on the trust level of your society. where the store resides.

The internet is a cesspool of vagrants, thieves, mentally unstable, people and software with no impulse control, pirates and that is just talking about corporations. It gets so much worse with individuals.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#162

Post by throw10920 »

bobosola wrote: post #72

I dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced. Also, inserting hidden or misleading links is specifically a no-no for Google Search [0], who have this to say: We detect policy-violating practices both through automated systems and,…

> I’m not convinced.

Is this how low we've sunk - that even below taking a single personal anecdote and generalizing it to everything - now we're taking zero experience and dismissing things based on vibes?

I've seen lots of LLM-slop-lovers doing the same thing. Maybe it's a pattern.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#163

Post by subscribed »

ImPostingOnHN wrote: post #130
spiderfarmer wrote: post #78

Earlier quoted context omitted.

[flagged]

Wow, how did you manually hand-write 6 million web pages? That is impressive. It would take me a while to even montonically count that high.

You're trying to use a quite unfunny "sarcasm" to move the goalpost to the strawman (they never claimed they handcrafted these pages) and quickly gloss ove the fact it's 20 years of work so why not?

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#164

Post by Bender »

snehesht wrote: post #12

Why not simply blacklist or rate limit those bot IP’s ?

Why not simply blacklist or rate limit those bot IP’s ?

Many bots cycle through short DHCP leases on LTE wifi devices. One would have to accept blocking all cell phones which I have done for my personal hobby crap but most businesses will not do this. Another big swath of bots come from Amazon EC2 and GoogleCloud which I will also happily block on my hobby crap but most businesses will not.

Some bots are easier to block as they do not use real web clients and are missing some TCP/IP headers making them ultra easy to block. Some also do not spoof user-agent and are easy to block. Some will attempt to access URL's not visible to real humans thus blocking themselves. Many bots can not do HTTP/2.0 so they are also trivial to block. Pretty much anything not using headless Chrome is easy to block.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#165

Post by subscribed »

gruez wrote: post #84
bengale wrote: post #67

Earlier quoted context omitted.

It’s interesting to see twists on the old anti-piracy arguments recycled for anti-ai.

Turns out many (most?) people on the internet were never anti-copyright in the first place. They were just anti-copyright (or at least, refused to challenge the anti-copyright people) because they wanted free movies and/or hated corporations.

Many of these people live int he countries where downloading for own use is lawful, since they're paying copyright levy exactly to cover for that.

They don't have to hate the copyright.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#166

Post by joquarky »

spiderfarmer wrote: post #78
pbasista wrote: post #49

Earlier quoted context omitted.

This is a dishonest analogy. In your example, there is only a limited amount of cookies available. While there is no practical limit on the amount of time a certain digital media can be viewed. You are allowed to take one cookie. But you are allowed to view a public website multiple times if you so want.

[flagged]

> If I can poison them and their families, I will.

Don't post anything online that you don't want to be brought up in court later.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#167

Post by subscribed »

aldousd666 wrote: post #35

This is ultimately just going to give them training material for how to avoid this crap. They'll have to up their game to get good code. The arms race just took another step, and if you're spending money creating or hosting this kind of content, it's not going to make up for the money you're losing by your other content getting scraped. The bottom has always been threatening to fall out of the ads paid for eyeballs,…

So, if at the end of the day instead of clicking EVERY single link in the repository they just check it out and parse locally...... I would consider it a win.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#169

Post by Forgeties79 »

obsidianbases1 wrote: post #106
Forgeties79 wrote: post #24

Earlier quoted context omitted.

Web crawlers didn’t routinely take down public resources or use the scraped info to generate facsimiles that people are still having ethical debates over. Its presence didn’t even register and it was indexing that helped them. It isn’t remotely the same thing. https://www.libraryjournal.com/story/ai-bots-swarm-library-c...

AI bots must've taken down that link you shared, it won't load :/ And search crawlers/results have been producing snippets that prevent users from clicking to the source for well over a decade. Edit: it loaded. I don't see how the problem isn't simply solved by an off the shelf solution like cloud flare. In the real world, you wouldn't open up a space/location if you couldn't handle the throughput. Why should online…

Why should everyone else pay the price for VC-funded, private companies? They should incur the cost.

This is no different than saying “robbers aren’t causing any problems, you just need to lock your doors, buy and set up sensors on every point of potential ingress, and pay a monthly cost for an alarm system. That’s on you.”

Post reply on HN