Apple’s new Transformer-powered predictive text model
1–10 of 268 posts
Re: Apple’s new Transformer-powered predictive text model
#2For text input support, being boring feels like a feature not a bug. For writing on mobile, you'll be writing a lot of boring short messages. That's said, it'd be cool if it can precondition with the app name - You'll probably write something more interesting on Pages vs on iMessage.
Re: Apple’s new Transformer-powered predictive text model
#3I was surprised to see the direct references to GPT2 by name.
Re: Apple’s new Transformer-powered predictive text model
#4[flagged]
Re: Apple’s new Transformer-powered predictive text model
#5And interesting test. I didn’t see a mention of the temperature setting used. Temperature controls the probability to pick a token that isn’t the top prediction, which leads to more creative/less robotic results.
For actual input prediction, you probably want the temperature to be zero. But even a model as good as GPT-3 becomes very boring and repetitive with those settings.
Re: Apple’s new Transformer-powered predictive text model
#6[flagged]
I know it's infuriating but that's how marketing works, and since they are a successful company, idk why they would stop using it. Just like some people say "RTX" instead of RayTracing now, it's quite a success for the company if it managed to replace a technical term in the mind of most consumers
Re: Apple’s new Transformer-powered predictive text model
#7[flagged]
[flagged]
Re: Apple’s new Transformer-powered predictive text model
#8[flagged]
What? Transformer model is a standard industry term for a particular machine learning model architecture: https://en.wikipedia.org/wiki/Transformer_(machine_learning_...
At 34M parameters, it's not very large. A LLM can be a transformer model, but also it can have a different architecture (e.g. RNN, such as RWKV).
Apple has been using RNNs for many things, I believe it's the first time they're shipping an on-device transformer model.
Re: Apple’s new Transformer-powered predictive text model
#9[flagged]
This is the one case where the technical term and Apple’s marketing actually do happen to overlap! These are transformer models. Apple didn't invent them, but their description of them is technically accurate. “LLM” is the dumbed-down term for the masses, describing only the “what” but not the “how”.
Re: Apple’s new Transformer-powered predictive text model
#10[flagged]
An LLM is a transformer model. That’s the standard, industry term. What are you going on about?