Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xerox scanners and photocopiers randomly alter numbers in scanned documents

dkriesel.com

Post reply on HN
Print view View original

71–80 of 118 posts

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#71

Post by linohh »

gngeal wrote: post #65
agl wrote: post #14

This class of error is called (by me, at least) a "contoot" because, long ago, when I was writing the JBIG2 compressor for Google Books PDFs, the first example was on the contents page of book. The title, "Contents", was set in very heavy type which happened to be an unexpected edge case in the classifier and it matched the "o" with the "e" and "n" and output "Contoots". The classifier was adjusted and these errors m…

How would one handle the case with the tiny boxes? It seems to me that these ought to be treated more like line drawings and not unify them as symbols at all if you can't properly decompose them into lines of Latin alphabet glyphs. JBIG2 of course cleverly doesn't tell you how to do the "smart" segmentation...

Yeah, and because the libraries are not open source, we'll never be able to check who failed big time.

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#72

Post by linohh »

scrumper wrote: post #70

I don't think it's necessarily an issue of inexcusable incompetence: it seems like one of those faults which is obvious in retrospect but very difficult to predict. Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers? That would seem to be a safer choice than writing a lossy compression algorithm from scratch. QA testing probably was on the order of 'picture looks right'; after all, why…

JBIG2 compression is in no way a standard compression algorithm, as the standard only describes decompression. The compression depends on the implementation. And this is where incompetence comes back into the game.

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#73

Post by linohh »

hga wrote: post #10
ElliotH wrote: post #3

I can't quite see the reason why you would lossily compress something when your machine's purpose is to duplicate things. Anyone got a reasonable reason for doing this?

Good point. Looking at a product page ( http://www.office.xerox.com/multifunction-printer/color-mult... ), I see that the first model mentioned is multifunction, it can "Copy, email, fax, print, [and] scan". So it sounds like there's one code path and it's seriously broken. I looked at the first settings page, and while it's in German I can see it's 200 DPI. There's no excuse for default lossy compression when you're…

CCITT Group 4 also known as Modified Modified Read is in no way something you'd be wanting to use now ever. I'm telling you why MMR sucks hard:

1. It's monochrome. No greyscale, no color. This works for text and lines, but nothing else. No big surprise, it was designed for Fax. But this makes CCITT G3 and G4 lossy.

2. It has no defined endianess. This adds another fault risk which you won't see coming as long as you're working on an isolated platform but can hit you in the nuts when you change hardware or software.

3. The data does not contain resolution or dimensional information, as well as no information about endianess This means that you have to rely on a container providing these informations. It could be TIFF, it could be PDF, it could be something an intern coded during coffee break. This is good on one hand, but evil on the other. Software is sold, saying CCITT G4 compression (a standard, after all) is used, while the data can be embedded in proprietary containers.

4. It's a 2D compression, meaning the compression is applied on a matrix of binary pixel data. As the standard does not specify the dimensions, you depend on another image container like TIFF to provide information. Because G4 removed EOL markers, there is no way to reconstruct image dimensions from the compressed data alone.

5. It's not exactly fault tolerant. Transmission errors can influence larger areas of the image up to making the picture totally unreadable. Flipped bits are not too critical, missing bits are, due to the 2D compression.

There are many excellent, fault tolerant, standardized Image formats ready to use for document processing and archiving, CCITT G4 isn't exactly one of them.

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#74

Post by rdtsc »

scrumper wrote: post #70

I don't think it's necessarily an issue of inexcusable incompetence: it seems like one of those faults which is obvious in retrospect but very difficult to predict. Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers? That would seem to be a safer choice than writing a lossy compression algorithm from scratch. QA testing probably was on the order of 'picture looks right'; after all, why…

> Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers?

I have never heard of JBIG2. I implemented JPEG2000 codecs from scratch, arithmetic coding compression and I have never heard of JBIG2. And here the are using and it others claiming it is just a standard run of the mill thing.

> That would seem to be a safer choice than writing a lossy compression algorithm from scratch.

Going out on a limb here, wouldn't the safest be to just not use a lossy codec at all or use something like JPEG?

> QA testing probably was on the order of 'picture looks right';

Sorry. This is the company whose name is the equivalent to the verb "to copy". If plugging in an obscure codec from some place and checking if one picture looks "OK" is their idea of QA then they deserve all the ridicule and lawsuits stemming from this.

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#75

Post by gridspy »

delinka wrote: post #28
thwest wrote: post #24

Earlier quoted context omitted.

Then realize you shouldn't define your life based on some digital records.

I don't define my life based on some digital records. But law enforcement (or the executive branch of the US Federal Government, including the NSA) does. And therein lies the problem. Someone will be convicted (perhaps even without the intervention of a court) based on unimpeachable but falsified digital records.

That sounds like the fly in the printer at the start of (the movie) Brazil.

Which of course leads to the conviction and torture of an innocent....

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#76

Post by rayiner »

scrumper wrote: post #70

I don't think it's necessarily an issue of inexcusable incompetence: it seems like one of those faults which is obvious in retrospect but very difficult to predict. Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers? That would seem to be a safer choice than writing a lossy compression algorithm from scratch. QA testing probably was on the order of 'picture looks right'; after all, why…

> Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers?

The problem isn't using a standard compression algorithm. It's failing to consider the properties of the algorithm used in relation to the problem domain.

A classic mistake engineering students make is to try and use familiar equations anywhere that the units work out. As a result, engineering professors hammer in the idea that before using any equation, you have to ask yourself: what are the assumptions underlying this equation, and do those assumptions hold for my specific problem? Similarly, if you're writing software for copiers, you should ask the basic question of whether a particular compression algorithm was appropriate for the particular types of images being compressed. It's incredibly basic.

I can totally see why this error happened. It was the equivalent of the engineering student blithely applying any equation where the units work out. Uncompressed pixels go in, compressed data comes out. Compression algorithms are substitutable... except when they're not.

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#77

Post by blt »

rly_ItsMe wrote: post #12
ElliotH wrote: post #3

I can't quite see the reason why you would lossily compress something when your machine's purpose is to duplicate things. Anyone got a reasonable reason for doing this?

In the good old days of analog copiers this would be impossible - the scanner send the light through a system of mirrors to the drum, the drum gets static charged, the toner is pulled on the charged parts and gets transferred to the transfer belt, here the paper has the opposite charge and pulls the toner off of the transfer belt, goes through the fusing unit and here is the toner 'burned' to the paper. End of Story…

Yup, and those old analog copiers - good ones at least - had beautiful crisp output. The resolution was good enough to reproduce printing dots so they could even duplicate photos from books. Continuous tone of an analog photograph didn't work as well. They sure were expensive though.

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#78

Post by hga »

linohh wrote: post #73
hga wrote: post #10

Earlier quoted context omitted.

Good point. Looking at a product page ( http://www.office.xerox.com/multifunction-printer/color-mult... ), I see that the first model mentioned is multifunction, it can "Copy, email, fax, print, [and] scan". So it sounds like there's one code path and it's seriously broken. I looked at the first settings page, and while it's in German I can see it's 200 DPI. There's no excuse for default lossy compression when you're…

CCITT Group 4 also known as Modified Modified Read is in no way something you'd be wanting to use now ever . I'm telling you why MMR sucks hard: 1. It's monochrome. No greyscale, no color. This works for text and lines, but nothing else. No big surprise, it was designed for Fax. But this makes CCITT G3 and G4 lossy. 2. It has no defined endianess. This adds another fault risk which you won't see coming as long as you…

Errr, I didn't communicate clearly.

What I meant to say was that CCITT Group IV gave acceptable sizes for early-'90s computing power, CPU and disk, and something at or better than its level of lossless compression today should be even more acceptable.

And in light of this screwup, I suspect we'd agree that Xerox would have been better off to use lossless (well, after the scanning, as you point out, but then again no one was willing to pay for color) CCITT Group IV than overly clever lossy JBIG2.

"It could be TIFF, it could be PDF, it could be something an intern coded during coffee break."

It could be something a journeyman software engineer edging to expect coded in a Saturday afternoon in a very fast paced project; for me, 3 weeks on the "engine". And, oh my, I can't remember encoding endiness, except of course for the leading TIFF bytes. But I had a guy who knew this cold telling me what to do, he was the one who debugged all our raw compressed data problems bit by bit. And, yeah, it was an "Intel" little endian TIFF, and I think I recall the Kodak Powescans produced that (600 pound monsters that could scan 18 inches per second at 200 DPI).

Hmmm, at least back then, "TIFF" was the selling point, and, oh yeah, it's Group IV compressed (except of course when it wasn't, we once dealt with some weird enhanced Group III).

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#79

Post by [deleted] »

rayiner wrote: post #76
scrumper wrote: post #70

I don't think it's necessarily an issue of inexcusable incompetence: it seems like one of those faults which is obvious in retrospect but very difficult to predict. Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers? That would seem to be a safer choice than writing a lossy compression algorithm from scratch. QA testing probably was on the order of 'picture looks right'; after all, why…

> Why shouldn't Xerox use a standard compression algorithm in their scanner/copiers? The problem isn't using a standard compression algorithm. It's failing to consider the properties of the algorithm used in relation to the problem domain. A classic mistake engineering students make is to try and use familiar equations anywhere that the units work out. As a result, engineering professors hammer in the idea that befor…

[deleted]

Re: Xerox scanners and photocopiers randomly alter numbers in scanned documents

#80

Post by ChuckMcM »

Given the challenges of JBIG2 it seems one should be able to construct a 'test' page which, when scanned, will test the algorithm's accuracy.

Once you have that, you can turn it into a sales too for folks selling Multi-function Printers such that there are "good" printers and "bad" printers, and then everyone will be forced to pass the test or be labeled a 'bad' printer.

Post reply on HN