Apple’s new Transformer-powered predictive text model • Hacker News Board

Re: Apple’s new Transformer-powered predictive text model

#201

Post by xp84 » Sun, Sep 17, 2023, 4:01 PM UTC

cubefox wrote: post #116

Zinu wrote: post #104

The example at the end made me wonder if Apple's model is actually better than GPT2 for text prediction. It generated garbage, but all that garbage made somewhat sense in the context of only the word "Today". Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly,…

The example at the end sounds just like the predictions you get from normal phone keyboards in the last couple of years, which presumably don't use a modern GPT-style language model. A bit disappointing.

Seriously disappointing. I was expecting that it would not produce total gibberish. It acts like it's a Markov chain, and only considers the last 1-2 words. Identical to the currently-shipping thing that we've had for the past however-many years.

Re: Apple’s new Transformer-powered predictive text model

#202

Post by mtkhaos » Sun, Sep 17, 2023, 4:09 PM UTC

brookst wrote: post #177

QuadrupleA wrote: post #167

Everyone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?

Most human communication is bland, and people who make a point of being unpredictable and shocking are usually pretty annoying. Think of it like spellcheck : the vast majority of the time it produces desired results, but if you really want to type bjPvc9fQ, you certainly can.

This throws out the possibility of high quality content.

Re: Apple’s new Transformer-powered predictive text model

#203

Post by avianlyric » Sun, Sep 17, 2023, 4:13 PM UTC

QuadrupleA wrote: post #167

Everyone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?

As someone with Dyslexia, who struggles a lot with written communication, and frequently finds myself fighting with spellcheckers, trying to get them to provide the correct correction. I’m very excited for these types of autocomplete systems. They’re like spellcheckers, except they also use context to produce better results, and frequently remove the need to play “guess the right misspelling” to get a normal spellcheck to provide good suggestions.

I’ll accept the risk of blandness, if it means that written communication finally becomes “easy” for me to participate in.

Re: Apple’s new Transformer-powered predictive text model

#204

Post by fulladder » Sun, Sep 17, 2023, 4:16 PM UTC

crazygringo wrote: post #178

QuadrupleA wrote: post #167

Everyone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?

It's not LLM's writing for us, it's just autocomplete. If the suggestion doesn't match what you were already planning on saying, you just ignore it. The human desire to be original and authentic is always going to be stronger. (It's much less effort to ignore it and keep typing your original thought, than it is to think about it, compare with what you were going to say, decide its version is better, and then accept i…

> The human desire to be original and authentic is always going to be stronger.

Authenticity didn't start to become a thing people cared about until about the early 1990s, and it didn't blow up and take over the mainstream culture until the 2000s and 2010s.

Prior to the 1990s, there was a lot of interest in professionalism. People spoke and wrote in overly formal, jargony ways that they perceived as being a marker for competence in some specialized professional domain. Being too honest/authentic would have been seen as unsophisticated and lower class.

Just pointing out that what people strive to emulate can change over time. Once LLM writing becomes commonplace, it will probably become trendy to write in the exact opposite way that an LLM does!

Re: Apple’s new Transformer-powered predictive text model

#205

Post by babl-yc » Sun, Sep 17, 2023, 4:21 PM UTC

QuadrupleA wrote: post #167

Everyone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?

I disabled Gmail Smart Compose for this reason; I felt like it was putting words into my mouth suggesting entire sentences for the email.

I'm much more open to using transformers as a better auto-correct, where it's one word at a time and uses the first letter as a filter. Especially on a tiny phone keyboard on the go.

Re: Apple’s new Transformer-powered predictive text model

#206

Post by avianlyric » Sun, Sep 17, 2023, 4:22 PM UTC

toxik wrote: post #124

dchest wrote: post #92

Earlier quoted context omitted.

Even large language models with billions of parameters get caught in the cycle. You don't usually see it exposed to users because there are sampling tricks applied, such as repetition/frequency penalty.

I don’t agree, it’s not very common for LLMs to get stuck in loops simply because loops are not commonly observed in the datasets.

That’s where you’re wrong. Raw LLM very often get stuck in loops, usually extremely small loops. There’s a big chunk of infrastructure that exists on the output end of any production LLM that exists explicitly for the purpose of preventing loops.

That post-processing infrastructure can use all kinds of mechanisms to prevent loops and induce more useful output. With the most basic simply systems simply refusing to select any output token that already in the input, to more complex stochastic process that explore the tree of possible outputs, to find branches whose overall result is improved by choosing less optimal immediate steps.

The vast majority of what make Chat GPT different to simpler GPT-3 models is this complex post-processing phase that allows designers to push and pull on the behaviour of the pre-baked static model underlying the chat interface.

Re: Apple’s new Transformer-powered predictive text model

#207

Post by tornato7 » Sun, Sep 17, 2023, 4:25 PM UTC

xp84 wrote: post #197

I want to know if this will be used to improve all the places where Apple devices attempt to interpret what you might mean to type, including the swipe keyboard. I've been suffering for years with their terrible, unusable swipe typing. You can't even get it to swipe "I love you" because it always prioritizes "your" over you, regardless of the context. I've even experimented extremely slowly and taken screen recording…

I love you I love you I love you I love you

Seems to work fine. Though I typically use Gboard because the swipe typing is much better.

I find the most restrictive thing about Apple's autocorrect and speech to text to be the limited vocabulary. Once you start using any industry terms it completely fails.

Re: Apple’s new Transformer-powered predictive text model

#208

Post by runjake » Sun, Sep 17, 2023, 4:26 PM UTC

QuadrupleA wrote: post #167

Everyone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?

Because at work I’m typing the same bland things all the time in documents and communications. I appreciate stuff like the predictive word stuff in Google Docs. It’s helpful because business language is expected to be normalized and boring.

On the other side of that token, the average language abilities of the average American office worker are pretty low so I’m assuming they view this as an enhanced AutoCorrect and they appreciate it because it makes them look less dumb.

I agree with your point though. And to answer your last question, unfortunately I think the answer is “yes”.

Re: Apple’s new Transformer-powered predictive text model

#209

Post by gorenb » Sun, Sep 17, 2023, 4:26 PM UTC

tornato7 wrote: post #207

xp84 wrote: post #197

I want to know if this will be used to improve all the places where Apple devices attempt to interpret what you might mean to type, including the swipe keyboard. I've been suffering for years with their terrible, unusable swipe typing. You can't even get it to swipe "I love you" because it always prioritizes "your" over you, regardless of the context. I've even experimented extremely slowly and taken screen recording…

I love you I love you I love you I love you Seems to work fine. Though I typically use Gboard because the swipe typing is much better. I find the most restrictive thing about Apple's autocorrect and speech to text to be the limited vocabulary. Once you start using any industry terms it completely fails.

why are you using a

tag

Re: Apple’s new Transformer-powered predictive text model

#210

Post by fulladder » Sun, Sep 17, 2023, 4:29 PM UTC

QuadrupleA wrote: post #175

elicash wrote: post #173

Earlier quoted context omitted.

"Bing Chat's" implementation already allows you to select more creative generation of text. It's just a radio button option. There are also different technical solutions for the LLM to select which word to generate that allow either for more interesting, or more predictable, words. This isn't to say a human element doesn't have a ton to offer! Just to say that we aren't necessarily engineering a bland world of pablum…

That's just temperature, which evens out the random probability a little of the N most probable next words. It's still vastly favoring the N most common ones based on the training corpus, and will have a hard time producing uncommon ones. E.g. try asking an LLM to name a real, non-famous person. The internet and it's training corpus is full of regular people, but you won't have much luck - they're statistically too u…

Early versions of GPT would tell me about myself, but July of this year it was saying that it could not comment on a private individual. I'm not at all famous, but there's plenty of writing by and about me in the common training datasets.

So, I don't agree that there's insufficient data for it to remember a random person. This was obviously a conscious decision, probably in response to situations like this one last spring: https://www.businessinsider.com/chatgpt-ai-made-up-sexual-ha...