Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faking a JPEG

ty-penguin.org.uk

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Faking a JPEG

#81

Post by sltkr »

jeroenhd wrote: post #64

This makes me wonder if there are more efficient image formats that one might want to feed botnets. JPEG is highly complex, but PNG uses a relatively simple DEFLATE stream as well as some basic filters. Perhaps one could make a zip-bomb like PNG that only consists of a few bytes?

DEFLATE has a rather low maximum compression ratio of 1:1032, so a file that would take 1 GB of memory uncompressed still needs to be about 1 MB.

ZIP bombs rely on recursion or overlapping entries to achieve higher ratios, but the PNG format is too simple to allow such tricks (at least in the usual critical chunks that all decoders are required to support).

Re: Faking a JPEG

#82

Post by thayne »

I'm curious how the author identifies the crawlers that use random User Agents and and distinct ip addresses per request. Is there some other indicator that can be used to identify them?

On a different note, if the goal is to waste resources for the bot, on potential improvement could be to uses very large images with repeating structure that compress extremely well as jpegs for the templates, so that it takes more ram and cpu to decode them with relatively little cpu and ram required to generate them and bandwidth to transfer them.

Re: Faking a JPEG

#83

Post by suspended_state »

andybak wrote: post #61
suspended_state wrote: post #54

Earlier quoted context omitted.

The point is that not every web crawler is out there to scrape websites.

Unless you define "scrape" to be inherently nefarious - then surely they are? Isn't the definition of a web crawler based on scraping websites?

I think that web scraping is usually understood as the act of extracting information of a website for ulterior self-centered motives. However, it is clear that this ulterior motive cannot be assessed by a website owner. Only the observable behaviour of a data collecting process can be categorized as morally good or bad. While the bad behaving people are usually also the ones with morally wrong motives, one doesn't entail the other. I chose to qualify the bad behaving ones as scrapers, and the good behaving ones as crawlers.

That being said, the author is perhaps concerned by the growing amount of collecting process, which carries a toll on his server, and thus chose to simply penalize them all.

Re: Faking a JPEG

#84

Post by superjan »

superjan wrote: post #36

There is a particular pattern (block/tag marker) that is illegal the compressed JPEG stream. If I recall correctly you should insert a 0x00 after a 0xFF byte in the output to avoid it. If there is interest I can followup later (not today).

Ok this is correct for traditional JPEG. Other flavors like Jpeg2000 use a similar (but lower overhead) version of this byte-stuffing to avoid JPEG markers from appearing in the compressed stream.

Related: https://en.wikipedia.org/wiki/JPEG#Syntax_and_structure

Re: Faking a JPEG

#85

Post by ethan_smith »

superjan wrote: post #36

There is a particular pattern (block/tag marker) that is illegal the compressed JPEG stream. If I recall correctly you should insert a 0x00 after a 0xFF byte in the output to avoid it. If there is interest I can followup later (not today).

You're referring to JPEG's byte stuffing rule: any 0xFF byte in the entropy-coded data must be followed by a 0x00 byte to prevent it from being interpreted as a marker segment.

Re: Faking a JPEG

#87

Post by darkwater »

gary_0 wrote: post #53
josephg wrote: post #51

Earlier quoted context omitted.

> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. So? What duty do web site operators have to be "nice" to people scraping your website?

The Marginalia search engine or archive.org probably don't deserve such treatment--they're performing a public service that benefits everyone, for free. And it's generally not in one's best interests to serve a bunch of garbage to Google or Bing's crawlers, either.

If you have such a website, then you will just serve normal data. But it seems perfectly legit to serve fake random gibberish from your website if you want to. A human would just stop reading it.

Re: Faking a JPEG

#88

Post by schroeding »

mhuffman wrote: post #80

I don't understand the reasoning behind the "feed them a bunch of trash" option when it seems that if you identify them (for example by ignoring a robots.txt file) you can just keep them hung up on network connections or similar without paying for infinite garbage for crawlers to injest.

The "poisoning the data supply" angle seems to be a common motive, similar to tools like nightshade[1] (for actual images and not just garbage data).

[1] https://nightshade.cs.uchicago.edu/whatis.html

Re: Faking a JPEG

#89

Post by DamonHD »

yorwba wrote: post #58
bstsb wrote: post #52

Earlier quoted context omitted.

previously the author wrote in a comment reply about not configuring robots.txt at all: > I've not configured anything in my robots.txt and yes, this is an extreme position to take. But I don't much like the concept that it's my responsibility to configure my web site so that crawlers don't DOS it. In my opinion, a legitimate crawler ought not to be hitting a single web site at a sustained rate of > 15 requests per s…

The spigot doesn't seem to distinguish between crawlers that make more than 15 requests per second and those that make less. I think it would be nicer to throw up a "429 Too Many Requests" page when you think the load is too much and only poison crawlers that don't back off afterwards.

Almost no bot responds usefully to 429 that I have seen, and a few respond to it like 500 and 503 to speed up / retry / poll more.

Re: Faking a JPEG

#90

Post by anarazel »

kuschku wrote: post #73
Szpadel wrote: post #55

the worst offender I saw is meta. they have facebookexternalhit bot (they sometimes use default python request user agent) that (as they documented) explicitly ignores robots.txt it's (as they say) used to validate links if they contain malware. But if someone would like to serve malware the first thing they would do would be to serve innocent page to facebook AS and their user agent. they also re-check every URL eve…

Since when is 10r/s flooding? That barely registers as a blip even if you're hosting your site on a single server.

That depends on what you're hosting. Good luck if it's e.g. a web interface for a bunch of git repositories with a long history. You can't cache effectively because there's too many pages and generating each page isn't cheap.
Post reply on HN