The next step would be to "depixelate" the resulting image. How could this be done? I guess OCR would not work because of the variation of the fonts (you don't want the document to end up in a single font; you want to keep the fonts). Could a deep learning approach work here, even if it has not been trained on all the specific fonts?
Page Dewarping
61–70 of 75 posts
Re: Page Dewarping
#62Recently, Dropbox wrote about dewarping prior to OCR in their app: https://news.ycombinator.com/item?id=12297944 This code had the same idea, and is open-source!
Dewarping images for OCR isn't a new idea at all - while older systems used to do simple de-skew (tilting the image) for speed reasons, the need for warping the image to improve results has been known for a very long time, and most production quality OCR engines has done something like it for a long time.
(this isn't to slight the article - it's a great, well written presentation on how to implement it)
Re: Page Dewarping
#63The next step would be to "depixelate" the resulting image. How could this be done? I guess OCR would not work because of the variation of the fonts (you don't want the document to end up in a single font; you want to keep the fonts). Could a deep learning approach work here, even if it has not been trained on all the specific fonts?
Plenty of engines will do OCR and use the shapes recognised with high certainty to affect how they detect the rest.
There are many ways of doing this, and you can achieve some results even without knowing if your image is text, but just has lots of self-similarity by virtually sliding a "grid" over the image, slicing it up into n-by-n squares, running any of a number of nearest-neighbour variants over it, and then for each cluster replace all instances of the squares in the cluster by the one which minimise the overall error rate vs the others.
This will work reasonably well for very structured images such as text, as long as enough characters are near correct, and will retain custom fonts etc. but clean them up quite a bit as long as they either are different enough, or occur often enough on a page to not get "corrected".
I'm sure there are better ways of doing this too - it's been a decade since I kept up with OCR research.
Re: Page Dewarping
#64it came in handy whenever a student emailed me their homework as a pile of JPEGs. Gotta admire this guy's resourcefulness—and patience. If I were a professor, I'd probably just reject the assignment outright if a student sent me a bunch of photos from their smartphone in lieu of a PDF or a "proper" scan. :)
Reject it, in the form of a picture of a handwritten note on a print-out of his jpegs.
Re: Page Dewarping
#65"You can see these are not exactly small optimization problems. The smallest one has 89 parameters in the model, and the largest has 600." Those are small optmization problems. These types of problems are solved in computer vision for hundreds of thousands of variables. His problem can be solved in real-time, not tens of seconds.
Compared to weather prediction, which is an optimisation problem involving hundreds of millions of variables, and occupies a hefty supercomputing cluster for a few hours.
Re: Page Dewarping
#66it came in handy whenever a student emailed me their homework as a pile of JPEGs. Gotta admire this guy's resourcefulness—and patience. If I were a professor, I'd probably just reject the assignment outright if a student sent me a bunch of photos from their smartphone in lieu of a PDF or a "proper" scan. :)
Reject it, in the form of a picture of a handwritten note on a print-out of his jpegs.
For full effect, put the jpeg print outs on a wooden table, stick the note to it, take a photo, put it in word and take a photo of the screen.
Re: Page Dewarping
#67I use Microsoft's "Office Lens" app on my Android phone all the time, like a "smart camera" which automatically squares off and white-balances each photo of a page (usually mail or forms filled in by hand). It can't handle warped pages though, so I hope they add something like this!
Yes, Office Lens is great for that. I recently switched to CamScanner, which has an even better algorithm for deskewing pages (take a pic of a receipt at an odd angle, it automatically transforms it to flat and rectangular). Pro version was on sale -- I'm not affiliated, just a happy customer and highly related to this article.
I have it as well, but use Office Lens or Google Drive (create a shortcut to a folder on home screen of android device and use it to scan into it).
The reason is CamScanner tried to upsell me to some monthly plan.
For all good advice on HN that you should build recurring revenue: It seriously annoys me when people tries to do that by demanding monthly payments for static features.
(Totally OK with selling license keys for new features etc.)
Re: Page Dewarping
#68Here's something that I think has not been done, but could be quite lucrative, building a high resolution scanner using the phone camera, multiple pictures and interpolation/noise removal. Most phone cameras these days have good resolutions, and you could technically take a 6x4 photo, divvy it to 3x3 grid and take close up photos, and have smart algorithms interpolate the pixels to form a single image with high res.…
I've used some free apps that turn phones into document scanners. Almost all banks I use have something like that embedded for check deposits. Maybe they're just doing deskew rather than dewarping though... loose documents aren't usually warped like book pages.
The next hard problem would be help with DIY book scanning. Like the camera could sit over my shoulder and detect when I've turned a page, then automatically take a picture of the new page. Then OCR kicks in, and conversion to EPUB, preserving graphics when pages have non-text elements. Mostly just feel like we can probably do better than the massive contraptions over at www.diybookscanner.org
Re: Page Dewarping
#69Earlier quoted context omitted.
Yes, Office Lens is great for that. I recently switched to CamScanner, which has an even better algorithm for deskewing pages (take a pic of a receipt at an odd angle, it automatically transforms it to flat and rectangular). Pro version was on sale -- I'm not affiliated, just a happy customer and highly related to this article.
I have it as well, but use Office Lens or Google Drive (create a shortcut to a folder on home screen of android device and use it to scan into it). The reason is CamScanner tried to upsell me to some monthly plan. For all good advice on HN that you should build recurring revenue: It seriously annoys me when people tries to do that by demanding monthly payments for static features. (Totally OK with selling license key…
Update: CamScanner is now less annoying than what I used to remember. I might actually be switching back.
Re: Page Dewarping
#70Earlier quoted context omitted.
She scans it in a scanner, but the music is often in large books that don't lay completely flat, so you still see the warping near the binding.
I remember when I was in my middle school orchestra, occasionally I'd get a piece of sheet music with this problem. If it was bad enough, it might even make it impossible to read the last few notes. If you could fix this for your wife, I'm sure her students will greatly appreciate it!
Or, someone might play those last few notes ritardando, since they are bunching together. :)