Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xerox responds to the recent character substitution issue

realbusinessatxerox.blogs.xerox.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Xerox responds to the recent character substitution issue

#51

Post by Dylan16807 »

Groxx wrote: post #43
ToothlessJake wrote: post #41

Earlier quoted context omitted.

What are you on about? What does it produce if not computer-readable characters? Computer illegible characters? Are you saying it cannot read from the dictionary it creates? Or from the characters it is later optically recognizing off that dictionary? Again from the JBIG2 wiki[1]: "Textual regions are compressed as follows: the foreground pixels in the regions are grouped into symbols. A dictionary of symbols is then…

Does it produce ASCII? UTF? If no, it's not OCR. edit: by the definition you seem to be going on, any facial recognition is also OCR, since you could consider a face a 'glyph' (edit: 'symbol'). The only 'text' thing here that I can see is that it is intended to be used on text, which lends some optimizations, nothing that it's actually text-based in any way.

If you make a font out of faces and use them as repeated glyphs then yes it's OCR. If you're not using identical symbols over and over than I don't think you have a sane definition of 'glyph'.

Re: Xerox responds to the recent character substitution issue

#52

Post by Dylan16807 »

eCa wrote: post #47
ToothlessJake wrote: post #41

Earlier quoted context omitted.

What are you on about? What does it produce if not computer-readable characters? Computer illegible characters? Are you saying it cannot read from the dictionary it creates? Or from the characters it is later optically recognizing off that dictionary? Again from the JBIG2 wiki[1]: "Textual regions are compressed as follows: the foreground pixels in the regions are grouped into symbols. A dictionary of symbols is then…

It produces symbols, not characters. Say that the scanner internally splits the scan into regions of 10x10 pixels that it saves in memory. If another region differs on less than (say) 10% of the pixels it is assumed that the two zones are identical and the first one is used in the second place too. The regions have no semantic meaning. OCR translates the scan into a character set.

The only thing that's missing is a mapping from 'symbol #28' into 'ascii #63'. Internally it's storing instances of symbols plus font data for those symbols.

Also, something to think about: an EBCDIC document accidentally printed as ASCII/8859-1 would have equally zero semantic meaning when fed into an OCR program. But I don't think anyone would argue it wasn't OCR.

Re: Xerox responds to the recent character substitution issue

#53

Post by wtallis »

hyborg787 wrote: post #6
wtallis wrote: post #4

So they claim that the fine print warns about character substitution. But they still are willing to label the option with that problem "normal quality" and suggest using "high quality" to get strictly image compression applied with no OCR. They don't seem to understand that a photocopier should in its normal operating mode never do post-processing that creates such surprising and misleading artifacts - better illegib…

This has nothing to do with OCR. It's an issue with the JBIG2 compression re-using similar patches as substitutes for certain areas of the images if they're "close enough". This issue is exacerbated at lower resolutions.

The process that's in use and what you consider to be true OCR are very similar, differing mainly in the last step (where OCR maps to a standard character set, but JBIG2 creates one on the fly along with a corresponding font). However, the errors at issue arise from part of the process where JBIG2 and OCR are doing pretty much the same thing, so even if the analogy is flawed, it is still highly instructive. Saying this has nothing to do with OCR is quite simply wrong, since this is clearly very closely related with OCR even if it doesn't meet your exact (unspecified) definition of OCR.

Re: Xerox responds to the recent character substitution issue

#54

Post by mansr »

mikeash wrote: post #33
gwright wrote: post #28

Earlier quoted context omitted.

But the scanner isn't starting off with "17" (as in two ascii characters) it is starting off with a bit mapped image that your brain happens to interpret as the number 17. It is too much to ask that a lossy image compression algorithm never result in a compressed bitmap that your brain interprets exactly as the original. Having various compression/quality options allows you to pick the tradeoff (file size/resulting q…

I don't believe that e.g. JPEG compression can ever take a clearly readable 17 and turn it into a clearly readable 21. Lossy compression implies changing the data, obviously, but it does not have to imply changing the data in that particular way.

JPEG will not do that, but MPEG at high compression levels might.

Re: Xerox responds to the recent character substitution issue

#55

Post by Groxx »

Dylan16807 wrote: post #52
eCa wrote: post #47

Earlier quoted context omitted.

It produces symbols, not characters. Say that the scanner internally splits the scan into regions of 10x10 pixels that it saves in memory. If another region differs on less than (say) 10% of the pixels it is assumed that the two zones are identical and the first one is used in the second place too. The regions have no semantic meaning. OCR translates the scan into a character set.

The only thing that's missing is a mapping from 'symbol #28' into 'ascii #63'. Internally it's storing instances of symbols plus font data for those symbols. Also, something to think about: an EBCDIC document accidentally printed as ASCII/8859-1 would have equally zero semantic meaning when fed into an OCR program. But I don't think anyone would argue it wasn't OCR.

That "only thing that's missing" is a very very big thing, and difficult to get correct. And where does it say it's storing font data for the symbols?

Re: Xerox responds to the recent character substitution issue

#56

Post by Dylan16807 »

Groxx wrote: post #55
Dylan16807 wrote: post #52

Earlier quoted context omitted.

The only thing that's missing is a mapping from 'symbol #28' into 'ascii #63'. Internally it's storing instances of symbols plus font data for those symbols. Also, something to think about: an EBCDIC document accidentally printed as ASCII/8859-1 would have equally zero semantic meaning when fed into an OCR program. But I don't think anyone would argue it wasn't OCR.

That "only thing that's missing" is a very very big thing, and difficult to get correct. And where does it say it's storing font data for the symbols?

A font doesn't need to be anything more than a series of bitmaps. And then each character location on the image, ignoring errors, references one of these bitmaps. That's how documents with embedded bitmap fonts generally work.

That mapping isn't a very big thing. Sometimes text-based PDFs don't even have it, and you don't notice unless you try to copy out and get the wrong letters.

Re: Xerox responds to the recent character substitution issue

#57

Post by duskwuff »

morsch wrote: post #50

I was curious to see how JBIG2 fares compared to JPEG, and found a benchmark from 2010 [1] comparing the file size of the resulting PDF: convert *.jpg JPEG.pdf -- 43777 kb convert *.png PNG.pdf -- 6907 kb jbig2 -b J -d -p -s *.jpg; pdf.py J > JBIG2.pdf -- 947 kb jbig2 -b J -d -p -s -2 *.jpg; pdf.py J > 2xJBIG2.pdf -- 1451 kb Quite a difference. I don't quite understand how JPEG fares so poorly compared to (lossless)…

JPEG is optimized for photographic images with lots of smooth gradients. It does badly with sharp edges, which scanned documents tend to contain a lot of.

Re: Xerox responds to the recent character substitution issue

#58

Post by glanch »

mikeash wrote: post #31

Standard idiot-box weasel-wording. Another case study to put on the enormous pile of examples of how not to communicate with your customers. Pretend you care, blame the users, and don't take any action. Hey, what could be wrong with that?

Just spitballing here, but if I'm Xerox, from a cost-benefit perspective, does it really make sense to do anything other than this?

They have millions of pieces of hardware in production across the world in all kinds of industries handling sensitive data, so the legal issues would overwhelm them if they responded to this with an apology and admittance of guilt of any kind. Sadly it just makes more sense to handle this the way they did. No it isn't the best thing for the users, but it's the best thing for Xerox. And that's how decisions get made most of the time.

Re: Xerox responds to the recent character substitution issue

#59

Post by DrStalker »

Gormo wrote: post #38
gwright wrote: post #28

Earlier quoted context omitted.

But the scanner isn't starting off with "17" (as in two ascii characters) it is starting off with a bit mapped image that your brain happens to interpret as the number 17. It is too much to ask that a lossy image compression algorithm never result in a compressed bitmap that your brain interprets exactly as the original. Having various compression/quality options allows you to pick the tradeoff (file size/resulting q…

It seems a bit too coincidental that images to which human beings assign semantic value are being transformed into images to which human beings assign different semantic value. I don't expect the scanner to have any semantic awareness of the document content, so when I hear "lossy compression", my expectation is "image may become illegible", and not "image may remain legible, but become inaccurate".

It's not a coincidence that the thing that looks most like a blurred number is another blurred number.

A document will be covered in numbers, and the compression algorithm looks for similar blocks it can re-use; the side effect is sometimes it says "that blurry 4 looks pretty close to this blurry two, so I'll just store that block once and reuse it"

The problem is that this is a minor side effect to a programmer and an absolutely massive issue to an end user that no-one had thought of previously, and now we all have to be worried that all our scanned documents might be incorrect. (just because this was found in fuji-xerox scanners doesn't mean other brands don't also have the issue)

Re: Xerox responds to the recent character substitution issue

#60

Post by jessedhillon »

Gormo wrote: post #38
gwright wrote: post #28

Earlier quoted context omitted.

But the scanner isn't starting off with "17" (as in two ascii characters) it is starting off with a bit mapped image that your brain happens to interpret as the number 17. It is too much to ask that a lossy image compression algorithm never result in a compressed bitmap that your brain interprets exactly as the original. Having various compression/quality options allows you to pick the tradeoff (file size/resulting q…

It seems a bit too coincidental that images to which human beings assign semantic value are being transformed into images to which human beings assign different semantic value. I don't expect the scanner to have any semantic awareness of the document content, so when I hear "lossy compression", my expectation is "image may become illegible", and not "image may remain legible, but become inaccurate".

This is hacker news -- I don't expect everyone to know how jbig2 or other compression scheme works. But before you insinuate that the scanner has semantic awareness of the document and is altering that meaning in a less-than-coincidental way, I would hope that you could have a cursory look at how such compression works.

The issue only involves small letters, because the compression scheme breaks up the image into patches and then tries to identify visually similar blocks and reuse them. Certain settings can allow for small blocks of text to be deemed identical, within a threshold, and thus replaced. That's all. Coincidence, not semantic awareness.

Hence the advisory notice to use a higher resolution -- smaller block sizes.

Post reply on HN