Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faking a JPEG

ty-penguin.org.uk

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Faking a JPEG

#91

Post by marginalia_nu »

gary_0 wrote: post #53
josephg wrote: post #51

Earlier quoted context omitted.

> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. So? What duty do web site operators have to be "nice" to people scraping your website?

The Marginalia search engine or archive.org probably don't deserve such treatment--they're performing a public service that benefits everyone, for free. And it's generally not in one's best interests to serve a bunch of garbage to Google or Bing's crawlers, either.

It's not really too big of a problem for a well-implemented crawler. You basically need to define an upper bound both in terms of document count and time for your crawls, since crawler traps are pretty common and have been around since the cretaceous.

Re: Faking a JPEG

#92

Post by oofbey »

kuschku wrote: post #73
Szpadel wrote: post #55

the worst offender I saw is meta. they have facebookexternalhit bot (they sometimes use default python request user agent) that (as they documented) explicitly ignores robots.txt it's (as they say) used to validate links if they contain malware. But if someone would like to serve malware the first thing they would do would be to serve innocent page to facebook AS and their user agent. they also re-check every URL eve…

Since when is 10r/s flooding? That barely registers as a blip even if you're hosting your site on a single server.

If you're serving static pages through nginx or something, then 10/sec is nothing. But if you're running python code to generate every page, it can add up fast.

Re: Faking a JPEG

#93

Post by dhosek »

DamonHD wrote: post #89
yorwba wrote: post #58

Earlier quoted context omitted.

The spigot doesn't seem to distinguish between crawlers that make more than 15 requests per second and those that make less. I think it would be nicer to throw up a "429 Too Many Requests" page when you think the load is too much and only poison crawlers that don't back off afterwards.

Almost no bot responds usefully to 429 that I have seen, and a few respond to it like 500 and 503 to speed up / retry / poll more.

Reminds me of a service I led the development on where we had to provide mocks for the front end to develop against as well as develop against mocks of an external service which wasn’t ready for us to use.

When we finally were able to do an end-to-end test, everything worked perfectly on the first try.

Except, the front end REST library, when given a 401 error when an incorrect auth code was sent, retried the request rather than reporting to the user that there was an error which meant that entering an incorrect auth code would lock the user out of their account immediately.

We ended up having to return all results with a 200 response regardless of the contents because of that broken library.

Re: Faking a JPEG

#94

Post by rfrey »

Domainzsite wrote: post #66

This is pure internet mischief at its finest. Weaponizing fake JPEGs with valid structure and random payloads to burn botnet cycles? Brilliant. Love the tradeoff thinking: maximize crawler cost, minimize CPU. The Huffman bitmask tweak is chef’s kiss. Spigot feels like a spiritual successor to robots.txt flipping you off in binary.

But where is my chocolate cupcake recipe?

Re: Faking a JPEG

#95

Post by vintermann »

Dwedit wrote: post #40
jandrese wrote: post #31

I wonder if you could mess with AI input scrapers by adding fake captions to each image? I imagine something like: (big green blob) "My cat playing with his new catnip ball". (blue mess of an image) "Robins nesting"

A well-written scraper would check the image against a CLIP model or other captioning model to see if the text there actually agrees with the image contents.

Ain't nobody got the processing time for that! Scraping is about more, more, more. If they do any filtering it'll be afterwards.

Re: Faking a JPEG

#96

Post by vintermann »

superjan wrote: post #84
superjan wrote: post #36

There is a particular pattern (block/tag marker) that is illegal the compressed JPEG stream. If I recall correctly you should insert a 0x00 after a 0xFF byte in the output to avoid it. If there is interest I can followup later (not today).

Ok this is correct for traditional JPEG. Other flavors like Jpeg2000 use a similar (but lower overhead) version of this byte-stuffing to avoid JPEG markers from appearing in the compressed stream. Related: https://en.wikipedia.org/wiki/JPEG#Syntax_and_structure

I remember there was a guy on compression forums who was very annoyed at this waste of coding space. If you're doing compression, shouldn't you make sure every encoded file decodes to a distinct output? He thought so, and made bijective versions of Huffman coding, arithmetic coding, LZ coding and (even more impressive) the BWT transform known from bzip2.

He was a bit crazy, but I liked that guy. Rest in peace, David A. Scott. Maybe there will be new uses for making all compression bijective over all byte streams.

Re: Faking a JPEG

#97

Post by mhuffman »

schroeding wrote: post #88
mhuffman wrote: post #80

I don't understand the reasoning behind the "feed them a bunch of trash" option when it seems that if you identify them (for example by ignoring a robots.txt file) you can just keep them hung up on network connections or similar without paying for infinite garbage for crawlers to injest.

The "poisoning the data supply" angle seems to be a common motive, similar to tools like nightshade[1] (for actual images and not just garbage data). [1] https://nightshade.cs.uchicago.edu/whatis.html

I get that, but paying to do it?
Post reply on HN