RNN-Based Handwriting Recognition in Gboard • Hacker News Board

Re: RNN-Based Handwriting Recognition in Gboard

#11

Post by perone » Fri, Mar 08, 2019, 3:17 PM UTC

reubenmorais wrote: post #5

The "Making it Work, On-device" paragraph makes it seem like TensorFlow Lite will easily get your model running fast on-device, but in reality RNNs aren't currently supported by the TFLite Converter and the TFLiteLSTMCell example is super slow for training, so this is actually based on proprietary code not available to mere mortals using open source TensorFlow. If you were to actually try reproducing this work, you'd…

That is an important point, Google is the master of releasing things in half, this was a common practice in Tensorflow since the initial release, they basically removed a lot of things to release it and it became a Frankenstein base of code. Bazel is another example, inside Google it works amazingly, but the open source project is a pain in the ass.

Re: RNN-Based Handwriting Recognition in Gboard

#12

Post by yorwba » Fri, Mar 08, 2019, 3:50 PM UTC

fxfan wrote: post #8

I used it on windows phone 5 years back for chinese- wonder if this is new to android? (on iPhone right now)

Handwriting recognition has been available on Android for a while (especially for Chinese). This article is "only" about incremental improvements.

Re: RNN-Based Handwriting Recognition in Gboard

#13

Post by geophertz » Fri, Mar 08, 2019, 4:17 PM UTC

I miss the time when in Gboard you could use the slide typing to type multiple words at once. This was so useful and made people like me who are unable to type quickly on a virtual keyboard (touch) to type very fast.

Re: RNN-Based Handwriting Recognition in Gboard

#14

Post by yorwba » Fri, Mar 08, 2019, 4:23 PM UTC

AlphaWeaver wrote: post #3

Really cool stuff! My phone isn't big enough to do handwriting on, so I'm not really sure where this is supposed to be used? On a tablet I guess?

I just tried tout for refirsttime, and although the keyboardspace on my phone is barely large enough to cran five characters in there, the input scrolls sideraysautomatically if you litt your finger long enoy gh. So longer words can be entered as well. It doesn't seem to reevaluate previously decoded segments baselon what follows, though, so you can end up with weird misspellings at the beyinning of words. I dont think! Im going to use it from now on, beaurthere cognition is balenow ghto requiresigniti cant editing and the frictonisatittooncomfortable vihout a stylus.

Edited with the QWERTY keyboard:

I just tried it out for the first time, and although the keyboard space on my phone is barely large enough to cram five characters in there, the input scrolls sideways automatically if you lift your finger long enough. So longer words can be entered as well. It doesn't seem to reevaluate previously decoded segments based on what follows, though, so you can end up with weird misspellings at the beginning of words. I don't think I'm going to use it from now on, because the recognition is bad enough to require significant editing and the friction is a bit too uncomfortable without a stylus.

Re: RNN-Based Handwriting Recognition in Gboard

#15

Post by dvh » Fri, Mar 08, 2019, 5:27 PM UTC

Isn't swiping inherently faster? With sweeping you need 1 angle (corner) for letter. Typical letter uses much more than 1 corner.

Re: RNN-Based Handwriting Recognition in Gboard

#16

Post by cbhl » Fri, Mar 08, 2019, 5:42 PM UTC

dvh wrote: post #15

Isn't swiping inherently faster? With sweeping you need 1 angle (corner) for letter. Typical letter uses much more than 1 corner.

Only if you're writing words in the dictionary, and you're using the English alphabet.

Handwriting recognition is way more impactful for users in, say, Chinese.

Re: RNN-Based Handwriting Recognition in Gboard

#17

Post by modeless » Fri, Mar 08, 2019, 6:16 PM UTC

Wow, it is really surprising to me that bezier curve control points produced by an optimization process would be good inputs to a neural net model. Small perturbations to the inputs could produce radically different bezier control points depending on the decisions made by the curve optimizer, so this forces the neural network to learn about the characteristics of the optimizer as well as the input.

Neural nets usually thrive on raw high dimensional inputs, so dramatically reducing the dimensionality of the input seems like a strange decision. I'm sure it improves speed, but I would expect higher accuracy by processing the raw input.

Re: RNN-Based Handwriting Recognition in Gboard

#18

Post by nielsbot » Fri, Mar 08, 2019, 7:09 PM UTC

modeless wrote: post #17

Wow, it is really surprising to me that bezier curve control points produced by an optimization process would be good inputs to a neural net model. Small perturbations to the inputs could produce radically different bezier control points depending on the decisions made by the curve optimizer, so this forces the neural network to learn about the characteristics of the optimizer as well as the input. Neural nets usuall…

I don't know ML, but I was surprised too, since converting a series of points into Bezier paths of a certain degree seems arbitrary...

Re: RNN-Based Handwriting Recognition in Gboard

#19

Post by jjtheblunt » Fri, Mar 08, 2019, 7:39 PM UTC

dajohnson89 wrote: post #7

I just switched from Android to iPhone, and Gboard on iPhone doesn't have the translation function. It also doesn't have multiple languages -- if I want to switch languages I have to exit out of Gboard and use the default iOS keyboard. Anyone know why these features for Gboard are missing on iOS?

these features are not missing: check settings

Re: RNN-Based Handwriting Recognition in Gboard

#20

Post by ocrcustomserver » Sat, Mar 09, 2019, 12:22 AM UTC

appleflaxen wrote: post #2

this is so awesome! but how is it that we have RNN solutions for handwriting when we don't even have a standard, canned RNN for OCR? I know tesseract and related projects exist, but when I've tried them they have been fairly brittle with lower accuracy than I was expecting. Accuracy was especially problematic for letter combinations like "-ing" that would consistently be recognized as "-mg". Is there a good ML OCR li…

The reason is that online OCR (this particular case) is entirely different from offline OCR.

Online OCR is when you input the strokes directly on the tablet/phone, so it becomes a sequence of XY coordinates with an associated timestamp. It takes into account where you start and where you end the stroke on the canvas, along with the intermediate points (information galore).

Offline OCR is when you take a photo of your handwriting in your notebook, so you just get the raw pixels of a image. In offline OCR, you'd also have to properly segment and binarize the image before the OCR step.

With that being said, tesseract (version 4) uses an LSTM.