Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple’s new Transformer-powered predictive text model

jackcook.com

Post reply on HN
Print view View original

111–120 of 268 posts

Re: Apple’s new Transformer-powered predictive text model

#111

Post by layer8 »

blindriver wrote: post #17

I don't want better predictive text. I need better autocorrect. Something happened about 6 years ago where the quality of the autocorrect fell off the roof, and it's been absolutely terrible since then. I spend too much of my time fighting with either mispelling, flipping to the wrong word even though I spelled the word properly, etc. It has made typing on my iPhone an unpleasant experience and I need it to change. T…

I don’t understand why people still put up with Apple’s autocorrect instead of just turning it off. It’s so wrong so often as to be utterly useless.

Re: Apple’s new Transformer-powered predictive text model

#112

Post by M4v3R »

layer8 wrote: post #111
blindriver wrote: post #17

I don't want better predictive text. I need better autocorrect. Something happened about 6 years ago where the quality of the autocorrect fell off the roof, and it's been absolutely terrible since then. I spend too much of my time fighting with either mispelling, flipping to the wrong word even though I spelled the word properly, etc. It has made typing on my iPhone an unpleasant experience and I need it to change. T…

I don’t understand why people still put up with Apple’s autocorrect instead of just turning it off. It’s so wrong so often as to be utterly useless.

Because it depends on the person. For me autocorrect is 99% spot on and so it’s a valid trade off for me to deal with correcting the remaining 1%. My wife on the other hand cannot live with it so she turned it off. And every time I type something on her phone I immediately feel slower because I know each of my key presses have to be much more deliberate.

Re: Apple’s new Transformer-powered predictive text model

#113

Post by basisword »

fragmede wrote: post #102
dkga wrote: post #53

Beyond fixing iPhone’s autocorrect as widely mentioned below, I wonder if in future versions Apple will end up switching to phi-1.5 or other models that are much smaller but trained on higher quality data. Would also be cool if they trained their own copilot for Xcode, given their obviously enturmes code base in Swift and Obj-C.

What's "high quality" refer to here? The amount of txt spk I use on my iphone keyboard, vs words that are in the dictionary, is heavily skewed in the direction of short words that a corpus consisting of English text is likely to omit.

You use text speak on an iPhone? How? I find it impossible due to autocorrect. Either it will correct with the full word or it will correct with a wrong word making it more efficient to just type in full.

Re: Apple’s new Transformer-powered predictive text model

#114

Post by stavros »

Zinu wrote: post #104

The example at the end made me wonder if Apple's model is actually better than GPT2 for text prediction. It generated garbage, but all that garbage made somewhat sense in the context of only the word "Today". Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly,…

It seems obvious to me that it's not, because if you asked a human to guess what comes after "today" in a text, they'd never say "probably some gibberish about a day a day".

Re: Apple’s new Transformer-powered predictive text model

#116

Post by cubefox »

Zinu wrote: post #104

The example at the end made me wonder if Apple's model is actually better than GPT2 for text prediction. It generated garbage, but all that garbage made somewhat sense in the context of only the word "Today". Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly,…

The example at the end sounds just like the predictions you get from normal phone keyboards in the last couple of years, which presumably don't use a modern GPT-style language model. A bit disappointing.

Re: Apple’s new Transformer-powered predictive text model

#117

Post by criddell »

pentae wrote: post #25
blindriver wrote: post #17

I don't want better predictive text. I need better autocorrect. Something happened about 6 years ago where the quality of the autocorrect fell off the roof, and it's been absolutely terrible since then. I spend too much of my time fighting with either mispelling, flipping to the wrong word even though I spelled the word properly, etc. It has made typing on my iPhone an unpleasant experience and I need it to change. T…

Same with voice to text translation. It's absolutely, utterly, worthless. I end up having to go and edit half the message. Apple can't even get basic things right

I use voice to text all the time on my iPhone and find it to be astonishingly good. I’m a 50 year old man who grew up in Southwestern Ontario so that gives you an idea of my accent.

Re: Apple’s new Transformer-powered predictive text model

#118

Post by cubefox »

dchest wrote: post #92
toxik wrote: post #70

The example output reads exactly like the existing output, I have had it get caught in exactly that cycle.

Even large language models with billions of parameters get caught in the cycle. You don't usually see it exposed to users because there are sampling tricks applied, such as repetition/frequency penalty.

Such penalties would only exist for fine-tuned models. Base models have only the temperature setting. As the example at the end shows, even GPT-2 seems "smarter" than the Apple model, probably because of the number of parameters.

Re: Apple’s new Transformer-powered predictive text model

#119

Post by kristopolous »

ComputerGuru wrote: post #35

Just give me the option of not having autocorrect change the previous word when I am typing a new one. I don’t move on to the next word unless the current one was correct or I have already corrected it. Why does it assume I’m stupid? It even does this when I’ve picked the completion for the prior word or swiped to type and then made manual corrections. It’s painfully obvious that it shouldn’t be touched again.

Apple has assumed its users are hapless morons for many years. Most people seem to prefer it this way.

Re: Apple’s new Transformer-powered predictive text model

#120

Post by tgv »

corbezzoli wrote: post #58
tgv wrote: post #55

Earlier quoted context omitted.

Modern language processing relies on pretty large corpora, and to bootstrap it requires a part to be of high quality and annotated, although for spelling correction you don't need annotation. Or you can go the GPT-3 route and use really huge corpora. If that doesn't exist for your language, and for many smaller languages it doesn't, it won't get better than 2007.

Indonesian is spoken by 200M people (45M of which as first language), I bet they could find some text for it. When joined with the Malaysian family, it brings the total speakers to 290M.

Building corpora has been an ongoing research activity since the 1990s. Many countries lacked the academic programs and support to build them. Often, the material isn't even there. E.g., wikipedia is a great source, but many "national" wikis are rather empty. The Bahasa wiki has 1/10th of the articles of the English one. They also seem to be shorter.

But also, Apple doesn't do much research. MS, however, has always had an active NLP research and support program, if only because of the importance of Word. But Apple depends on the research available from third parties.

Post reply on HN