Ferret: A Multimodal Large Language Model • Hacker News Board

Re: Ferret: A Multimodal Large Language Model

#141

Post by ben_w » Sat, Dec 23, 2023, 7:09 PM UTC

spookthesunset wrote: post #136

ben_w wrote: post #130

Earlier quoted context omitted.

I'd be (pleasantly) surprised; look at how long it takes them to allow system apps to get replaced with downloadable 3rd party. And even then, the activation keyword is likely to be whatever Apple says. Similar logic as 3rd party keyboards, don't want user input to get stuck on even merely potentially untrustworthy or buggy code.

Correct me if I’m wrong but isn’t the activation word “special” in that it has to work in very low power states? I always assumed that is why the wake words are fixed, because said words need to “fit” within a very small power budget.

Could be — I've heard rumours along those lines, but none came with evidence.

Re: Ferret: A Multimodal Large Language Model

#142

Post by devinprater » Sat, Dec 23, 2023, 7:13 PM UTC

They're already going multi-modal? Holy crap, if google can't deliver in the accessibility space for this (image descriptions better than "the logo for the company"), then I'll definitely go back to Apple. I mean I do hope Apple cleans out bugs and makes VoiceOver feel like it won't fall over if I breathed hard, but their image descriptions, even without an LLM, are already clean and clear. More like "A green logo on a black background", where Google is, like I said, more like "The logo for the company." I guess it's kinda what we get when AI is crowdsourced rather than given good, high quality data to work with.

Re: Ferret: A Multimodal Large Language Model

#143

Post by jonplackett » Sat, Dec 23, 2023, 7:13 PM UTC

Presumable because this is Conda none of this can be run on any Apple hardware despite people managing to get M processors to do a bit of dabbling with AI?

Re: Ferret: A Multimodal Large Language Model

#144

Post by _visgean » Sat, Dec 23, 2023, 7:16 PM UTC

jonplackett wrote: post #143

Presumable because this is Conda none of this can be run on any Apple hardware despite people managing to get M processors to do a bit of dabbling with AI?

> because this is Conda none of this can be run on any Apple hardware

conda supports m1? https://www.anaconda.com/blog/new-release-anaconda-distribut...

Re: Ferret: A Multimodal Large Language Model

#145

Post by jachee » Sat, Dec 23, 2023, 7:17 PM UTC

dontlaugh wrote: post #115

hmottestad wrote: post #84

Earlier quoted context omitted.

With iOS 17 they added a tiny little LLM to the predictive typing. I have the newest and greatest iPhone but I feel that I very rarely see it in action. I must assume that it’s just too slow at to keep up with my typing at the moment. Or it’s just not large enough to give very many useful suggestions.

It’s probably why autocomplete got drastically worse, to the point I’m considering turning it off entirely. Most “AI” features are so incredibly fragile they’re not worth deploying.

Autocomplete got worse because the new system in iOS17 didn’t retain training data from prior versions. It reset everyone back to untrained. I’ve been manually correcting specialized language I use daily (e.g. “iRacing”) on my 12 (non-pro) since iOS17 release, and now it gets it correct 99.5% of the time.

So, rather than turning it off, manually correct the incorrect completions and use the suggested words bar frequently and it will learn how you type. It’s just having to start over after tossing out several OSes worth of training that makes it feel worse.

Re: Ferret: A Multimodal Large Language Model

#146

Post by darthrupert » Sat, Dec 23, 2023, 7:23 PM UTC

behnamoh wrote: post #62

aaronbrethorst wrote: post #33

Earlier quoted context omitted.

Rumors suggest they’re gearing up to make iOS 18 an AI focused release. It’ll be interesting to see if they offer different capabilities for online/offline scenarios, or if their offerings are strictly offline. Here’s one story to offer some context. There are others. https://archive.is/en3VL

> Rumors suggest they’re gearing up to make iOS 18 an AI focused release. Don't underestimate Apple at disappointing enthusiasts like you and me. We've been hearing many awesome stories about the next thing Apple will do, only to realize their marketing team chose to keep it for future iOS/MBP/iPhone generations to keep the profits high.

My expectation of Apple is that they lurk in the shadows, looking at what others do while perfecting their own thing. Then on the day of release, they'll be a decade ahead of competition.

They've done this a dozen times already.

Re: Ferret: A Multimodal Large Language Model

#147

Post by darthrupert » Sat, Dec 23, 2023, 7:24 PM UTC

ben_w wrote: post #130

behnamoh wrote: post #63

Earlier quoted context omitted.

I hope they get rid of it completely. People on r/locallama and others have made much better assistants using GPT which use iOS APIs to control the phone. It's ridiculous that Apple still hasn't done anything useful regarding Siri.

I'd be (pleasantly) surprised; look at how long it takes them to allow system apps to get replaced with downloadable 3rd party. And even then, the activation keyword is likely to be whatever Apple says. Similar logic as 3rd party keyboards, don't want user input to get stuck on even merely potentially untrustworthy or buggy code.

That's not due to incompetence; that's due to not seeing any reason to do it.

Re: Ferret: A Multimodal Large Language Model

#148

Post by [deleted] » Sat, Dec 23, 2023, 7:27 PM UTC

tambourine_man wrote: post #58

hhh wrote: post #48

Earlier quoted context omitted.

Why would they dogfood Apple Silicon for training models? Seems like a waste of developer time to me.

Apple doesn’t even sell NVidia cards on their Mac Pros. Are they training it on Linux? I think Apple would strive to be great at all computing related tasks. “Oh, Macs are not good for that, you should get a PC” should make them sad and worried. AI/LLM is the new hot thing. If people are using Windows or Linux, you’re loosing momentum, hearts and minds… and sales, obviously.

[deleted]

Re: Ferret: A Multimodal Large Language Model

#149

Post by devinprater » Sat, Dec 23, 2023, 7:28 PM UTC

CaptainOfCoit wrote: post #8

Maybe the abstract of the paper is a better introduction to what this is: > We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates d…

This is going to be great for accessibility! Imagine being blind and loading up a video game and using this to figure out what's around, having everything described locally. I mean, um, well that's what I'd use it for anyway. But knowing Apple, we won't be able to prompt the LLM directly so that probably won't happen until 5 years from now.

Re: Ferret: A Multimodal Large Language Model

#150

Post by bjtitus » Sat, Dec 23, 2023, 7:28 PM UTC

causal wrote: post #74

tambourine_man wrote: post #58

Earlier quoted context omitted.

Apple doesn’t even sell NVidia cards on their Mac Pros. Are they training it on Linux? I think Apple would strive to be great at all computing related tasks. “Oh, Macs are not good for that, you should get a PC” should make them sad and worried. AI/LLM is the new hot thing. If people are using Windows or Linux, you’re loosing momentum, hearts and minds… and sales, obviously.

Don't think that follows. I doubt all their cloud services run on Apple hardware. They make consumer devices.

This

They ceded the data center environment years ago.