Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faking a JPEG

ty-penguin.org.uk

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Faking a JPEG

#51

Post by josephg »

tomsmeding wrote: post #43

They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. [1]: https://www.ty-penguin.org.uk/robots.txt [2]: https://www.ty-pen…

> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice.

So? What duty do web site operators have to be "nice" to people scraping your website?

Re: Faking a JPEG

#52

Post by bstsb »

tomsmeding wrote: post #43

They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. [1]: https://www.ty-penguin.org.uk/robots.txt [2]: https://www.ty-pen…

previously the author wrote in a comment reply about not configuring robots.txt at all:

> I've not configured anything in my robots.txt and yes, this is an extreme position to take. But I don't much like the concept that it's my responsibility to configure my web site so that crawlers don't DOS it. In my opinion, a legitimate crawler ought not to be hitting a single web site at a sustained rate of > 15 requests per second.

Re: Faking a JPEG

#53

Post by gary_0 »

josephg wrote: post #51
tomsmeding wrote: post #43

They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. [1]: https://www.ty-penguin.org.uk/robots.txt [2]: https://www.ty-pen…

> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. So? What duty do web site operators have to be "nice" to people scraping your website?

The Marginalia search engine or archive.org probably don't deserve such treatment--they're performing a public service that benefits everyone, for free. And it's generally not in one's best interests to serve a bunch of garbage to Google or Bing's crawlers, either.

Re: Faking a JPEG

#54

Post by suspended_state »

josephg wrote: post #51
tomsmeding wrote: post #43

They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. [1]: https://www.ty-penguin.org.uk/robots.txt [2]: https://www.ty-pen…

> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. So? What duty do web site operators have to be "nice" to people scraping your website?

The point is that not every web crawler is out there to scrape websites.

Re: Faking a JPEG

#55

Post by Szpadel »

the worst offender I saw is meta.

they have facebookexternalhit bot (they sometimes use default python request user agent) that (as they documented) explicitly ignores robots.txt

it's (as they say) used to validate links if they contain malware. But if someone would like to serve malware the first thing they would do would be to serve innocent page to facebook AS and their user agent.

they also re-check every URL every month to validate if this still does not contain malware.

the issue is as follows some bad actors spam Facebook with URLs to expensive endpoints (like some search with random filters) and Facebook provides then with free ddos service for your competition. they flood you with > 10 r/s for days every month.

Re: Faking a JPEG

#56

Post by BubbleRings »

Is there reason you couldn’t generate your images by grabbing random rectangles of pixels from one source image and pasting it into a random location in another source image? Then you would have a fully valid jpg that no AI could easily successfully identify as generated junk. I guess that would require much more CPU than your current method huh?

Re: Faking a JPEG

#58

Post by yorwba »

bstsb wrote: post #52
tomsmeding wrote: post #43

They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. [1]: https://www.ty-penguin.org.uk/robots.txt [2]: https://www.ty-pen…

previously the author wrote in a comment reply about not configuring robots.txt at all: > I've not configured anything in my robots.txt and yes, this is an extreme position to take. But I don't much like the concept that it's my responsibility to configure my web site so that crawlers don't DOS it. In my opinion, a legitimate crawler ought not to be hitting a single web site at a sustained rate of > 15 requests per s…

The spigot doesn't seem to distinguish between crawlers that make more than 15 requests per second and those that make less. I think it would be nicer to throw up a "429 Too Many Requests" page when you think the load is too much and only poison crawlers that don't back off afterwards.

Re: Faking a JPEG

#59

Post by Someone »

Dwedit wrote: post #40
jandrese wrote: post #31

I wonder if you could mess with AI input scrapers by adding fake captions to each image? I imagine something like: (big green blob) "My cat playing with his new catnip ball". (blue mess of an image) "Robins nesting"

A well-written scraper would check the image against a CLIP model or other captioning model to see if the text there actually agrees with the image contents.

Do scrapers actually do such things on every page they download? Sampling a small fraction of a site to check how trustworthy it is, I can see happen, but I would think they’d rather scrape many more pages than spend resources doing such checks on every page.

Or is the internet so full of garbage nowadays that it is necessary to do that on every page?

Re: Faking a JPEG

#60

Post by evgpbfhnr »

yorwba wrote: post #58
bstsb wrote: post #52

Earlier quoted context omitted.

previously the author wrote in a comment reply about not configuring robots.txt at all: > I've not configured anything in my robots.txt and yes, this is an extreme position to take. But I don't much like the concept that it's my responsibility to configure my web site so that crawlers don't DOS it. In my opinion, a legitimate crawler ought not to be hitting a single web site at a sustained rate of > 15 requests per s…

The spigot doesn't seem to distinguish between crawlers that make more than 15 requests per second and those that make less. I think it would be nicer to throw up a "429 Too Many Requests" page when you think the load is too much and only poison crawlers that don't back off afterwards.

when crawlers use a botnet to only make one request per ip per long duration that's not realistic to implement though..
Post reply on HN