Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compressing and enhancing hand-written notes (2016)

mzucker.github.io

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Compressing and enhancing hand-written notes (2016)

#51

Post by goerz »

ramses0 wrote: post #49
goerz wrote: post #41

In terms of compression for scanned notes, I haven't found anything that comes close to what even an older version of Adobe Acrobat yields, due to the use of the JBIG2 codec. Has anybody found any way to compress PDF files with JBIG2 on Linux/Mac? It's pretty much the only reason I have to find a Windows machine with Acrobat installed a couple of times a year, to postprocess a batch of scanned PDFs.

`German and Swiss regulators have subsequently (in 2015) disallowed the JBIG2 encoding in archival documents.[19]`

Yeah, but I'm not archiving for the German or Swiss government. For scanned handwritten notes, JBIG2 still beats out anything else by an order of magnitude at least

Re: Compressing and enhancing hand-written notes (2016)

#52

Post by nekopa »

inetknght wrote: post #35
reaperducer wrote: post #22

Great job with that. I've only just started taking notes by hand once again, after being keyboard-only for many years. In your scenario, since you have assigned "scribes" taking the notes, you might be able to streamline the process with a "smart pen." There are several on the market. The one I got as a hand-me-down from a family member lets you write dozens of pages of notes, then Bluetooth them to a smartphone app…

I have found that my Galaxy Note 2014 is pretty much hands-down the best note taking tablet in my opinion. It's better than the crap that Microsoft and Apple are trying to hawk off. It doesn't have as many fancy apps but for _strictly_ note taking, sharing notes via email, and book reading, it's pretty awesome. I just wish its price would come down. It's still full price from four years ago :| and even getting more e…

I have to say that I love the galaxy note 8.0 for note taking. So much so that I'm trying to buy up as many as I can find so that I will always have one. There are 2 main reasons for this:

1st: I love the size. The 10s are a little too big to cart around in a jacket pocket, and the note phones are too small to be a decent notebook. (I must admit though, I'm rethinking this point, as I used to be a huge moleskin fan)

2nd: The Samsung s-note app from that time is awesome! Honestly, every time samsung comes out with a new pen based tablet or phone, I always check it out. And I'm looking for 2 features: a: the ability to import a pdf file & b: goddamned smart shapes! I suck at drawing, and so I love that on my 8.0 I can draw a shitty square or circle and the software will make it look all ship shape. But they seemed to have removed that feature from newer versions of s-note, and I haven't found a 3rd party app that can do just those 2 things.

Any pointers to an app that will allow me to import pdfs and draw on them using smart shapes would be awesome!

Re: Compressing and enhancing hand-written notes (2016)

#53

Post by pwg »

dunham wrote: post #42
trurl42 wrote: post #7

This reminds me of my time in university, when I saved all my lecture notes as DjVu [1] files. It's a great file format for space-efficient archiving of scans like that, with a bit of scripted preprocessing. [1]: https://en.wikipedia.org/wiki/DjVu

I like the idea, but DjVu seems to be very proprietary / single vendor and not in widespread use. This has made me reluctant to use it for archival purposes (vs say PDF, which has its own issues, but feels slightly more future proof to me). I think PDF can cover pretty much the same ground with JBig and Jpeg2k. (And I believe archive.org is doing that.) But I don't know of any open source code to do the segmentation…

There is an open-source DjVu library as well:

http://djvu.sourceforge.net/

Whether that makes the format "widespread" enough for your use case is of course your decision to make.

Re: Compressing and enhancing hand-written notes (2016)

#54

Post by WalterGR »

reaperducer wrote: post #22

Great job with that. I've only just started taking notes by hand once again, after being keyboard-only for many years. In your scenario, since you have assigned "scribes" taking the notes, you might be able to streamline the process with a "smart pen." There are several on the market. The one I got as a hand-me-down from a family member lets you write dozens of pages of notes, then Bluetooth them to a smartphone app…

Which smart pen do you have?

Re: Compressing and enhancing hand-written notes (2016)

#55

Post by [deleted] »

fouc wrote: post #40
kazinator wrote: post #29

I can get seemingly comparable results with a couple of simple operations in Gimp. Here is a casual job on the first image: https://i.imgur.com/Sy2rvsU.png The steps: 1. Duplicate the layer. 2. Gaussian-blur the top layer with big radius, 30+. 3. Put the top layer in "Divide" mode. Now the image is level. 4. Merge the layers together into one. 5. Use Color->Curves to clean away the writing bleeding through from the o…

You can get the higher resolution one here: https://github.com/mzucker/noteshrink/blob/master/examples/n... BTW I'm curious how you'd fare on the graph one. I didn't like his results for it. https://github.com/mzucker/noteshrink/blob/master/examples/g...

[deleted]

Re: Compressing and enhancing hand-written notes (2016)

#56

Post by wmu »

pjc50 wrote: post #3

Nice to see a bit of k-means clustering. I was worried that this might attempt to be "smart" by converting to symbols, replicating the "Xerox changes numbers in copied documents" bug, but it's pure pixel image processing. Very clean results. In some ways it's a smarter version of the "posterize" feature.

Here is the link to the blogpost that describes the problem: http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...?

Re: Compressing and enhancing hand-written notes (2016)

#57

Post by kazinator »

fouc wrote: post #40
kazinator wrote: post #29

I can get seemingly comparable results with a couple of simple operations in Gimp. Here is a casual job on the first image: https://i.imgur.com/Sy2rvsU.png The steps: 1. Duplicate the layer. 2. Gaussian-blur the top layer with big radius, 30+. 3. Put the top layer in "Divide" mode. Now the image is level. 4. Merge the layers together into one. 5. Use Color->Curves to clean away the writing bleeding through from the o…

You can get the higher resolution one here: https://github.com/mzucker/noteshrink/blob/master/examples/n... BTW I'm curious how you'd fare on the graph one. I didn't like his results for it. https://github.com/mzucker/noteshrink/blob/master/examples/g...

https://imgur.com/a/dsLhk

Note how the grid is completely gone, the Sharpie strokes are fuller and the ghosting around the red ink is gone. (The word "Red" seems to have been written faintly, like with a non-working ball point pen, and then written over properly.)

The thing is, I took a completely different approach here. I won't give a complete step-by-step recipe, but the gist of it is this:

1. Create a copy layer of the image.

2. Optionally level the intensity with the divide trick; I didn't bother.

3. Convert this copy to grayscale.

4. Threshold it to black and white, such that the grid is eliminated, but the writing remains solid.

5. Blur the writing (radius 3-4).

6. Threshold again.

7. Now you have a black and white version that is a bit thicker than the original. TURN THIS INTO A LAYER MASK. An inverted one which passes through the writing, and renders everything else transparent.

8. Apply this mask to the original image. This requires transfering a layer mask between layers.

9. Slide a white background under the masked layer. Now you have the lettering clean on white.

10. Play with simple Color->Brightness-Contrast. I ended up with something like brightness -66, contrast +88.

In the final step, because of the layer mask that is in effect, these controls affect only the writing: the white coming from the unaffected layer below stays white no matter what you do with the contrast and brightness controls.

Why the different approach: I first tried the original approach and the result was good. But I thought you wouldn't like it either. It was similar to Matt's. I did a better job of eliminating the grid, but the writing was less vivid. (Likewise I also preserved the yellow tint of the paper.) I wanted the grid completely gone, with vivid writing. Playing with the intensity transfer curves was not quite doing it; there was poor separation between vanshing the grid while preserving the ink.

This attempt can be seen here: https://imgur.com/a/ldrBN The green writing is particularly unsatisfactory.

Re: Compressing and enhancing hand-written notes (2016)

#58

Post by bb88 »

pfalke wrote: post #30
John_KZ wrote: post #20

Earlier quoted context omitted.

Is this really standard writing paper? I assume it would be useful for calligraphy or learning how to write (as you can use the subdivision to draw letters to the correct height) but I find it weird for it to be standard issue paper.

It is. It's called "French-ruled paper" and is the school standard in France. [0] [0] https://getfrenchbox.com/what-is-french-ruled-paper-seyes-sc...

My handwriting in school would have been better if I could have used this.

Re: Compressing and enhancing hand-written notes (2016)

#59

Post by derefr »

dtech wrote: post #19
alister wrote: post #17

I have an observation about scanning documents that results in good quality and smaller files, but I can't satisfactorily explain why it works. Consider these two cases: (1) Scan document at very high resolution as a JPG and then use a third-party program (like Photoshop or whatever) to re-encode the JPG at your preferred low resolution. (2) Scan document at your preferred low resolution as a JPG straight away. Don't…

> My theory is that scanner manufacturers are deliberately choosing the JPG encoding profile that gets them the fastest result. This is more-or-less correct. The chips in the printers have a lot less power than your CPU, and the algorithms are a lot worse than those in Photoshop.

I'm surprised there aren't any high-quality JPEG encoder ASICs.

Would scanners benefit from just using some of the plentiful, cheap, excellent-quality H.264 encoder ASICs—and treating the output as HEIF?

Re: Compressing and enhancing hand-written notes (2016)

#60

Post by inetknght »

WalterGR wrote: post #45
inetknght wrote: post #35

Earlier quoted context omitted.

I have found that my Galaxy Note 2014 is pretty much hands-down the best note taking tablet in my opinion. It's better than the crap that Microsoft and Apple are trying to hawk off. It doesn't have as many fancy apps but for _strictly_ note taking, sharing notes via email, and book reading, it's pretty awesome. I just wish its price would come down. It's still full price from four years ago :| and even getting more e…

Why is the Galaxy Note 2014 good for note taking? What app(s) do you use for note taking? What's bad about Microsoft's and Apple's offerings?

As far as Apple's offerings: no idea. The last Apple device I used was an 4th-generation iPod back when that was the new thing. Coming from a desktop-power-user background, iOS 4's UX was extremely offensive. So is Android's, by the way, but at least Android's horrid UX cheaper.

Technicalities: 1) the S Pen's sensitivity is much closer to the image. Microsoft Surface, in contrast, feels and appears about 1/4" above the image: that causes me to mis-write a lot since I feel the stylus at a different location from where the drawing is being applied. I don't know about you, but I don't write with my pen/pencil/stylus straight up 90 degrees from the surface being written upon which is the only way I could reliably use a Surface for writing.

2) the Galaxy Note 2014's resolution is higher than the Surfaces I've used even though it's on a smaller form factor. That means that the image simply looks sharper.

3) the S Pen uses a different tech so that the Galaxy Note is able to recognize the difference between my fingers (or my palm) resting on the tablet's surface vs the stylus writing whatever's being written.

4) Both of the Galaxy Note 2014s I've used have had a battery life of about a full week from a full charge. It does depend on how much you use it, of course, but typically it's about 4.5 days, dying right at the tail end of Friday. That's with writing notes for about 2 or 3 hours during every workday. Contrast with Surface 3 and Surface Pro ... one had lasted a day and a half on average while the other lasted about 10 hours.

5) S Note translates my handwriting with about a 90% accuracy. But OneNote translates my handwriting to literal garbage text for about 50% of the time. When it doesn't translate to garbage text, there's of course the common typos from thinking I->l->1 B->0->O->o->D->etc (which S Note also has trouble with, but not as much).

Opinions: I absolutely despite Microsoft's and Apple's OS offerings these days. I don't like Android, either, but it's the least hated of the three. I use Windows 7 at home for games but otherwise Linux the whole way. Linux allows me to actually customize the UI the way I want; when it doesn't, there's the good ole' command line.

I've found the S Note app to be fairly intuitive and straightforward. It's really easy to export notes to PDF or email, eg to share with coworkers. If you can get around OneNote's shitfest version of OCR, I'll admit that the rest of it pretty intuitive. I really didn't care to learn (yet another) different way of doing things though.

If I had _one_ complaint about the Galaxy Note, it's that I have yet to find a way to have it require Active Directory to unlock; so if that's a requirement then ~~your company sucks~~ you're SOL for using it at work. If someone knows of a viable way to login to an Active Directory account on Android, that'd be awesome.

I also have to say that I actively use Google Play Books. Good luck getting that to work on Surface or iOS outside of ~~Chrome~~ a browser; I never could anyway. If I have to log in to a site using a browser and use the browser to use the site instead of using native apps, then why not just use a Chromebook?

Post reply on HN