Ferret: A Multimodal Large Language Model • Hacker News Board

Re: Ferret: A Multimodal Large Language Model

#171

Post by skygazer » Sat, Dec 23, 2023, 8:05 PM UTC

kossTKR wrote: post #155

CaptainOfCoit wrote: post #138

Earlier quoted context omitted.

Yes, Siri is really dumb. But so is every 7B/13B model out there too.

Eh no, 7B Mistral / Deepseek would certainly almost already be able to function as a super Siri, but probably something closer to PHI-2 + the new MLX apple optimisations. Have you tried those? https://twitter.com/awnihannun/status/1735355067673526360 If trained on an iPhone API + documentation and given a little web access it would blow absolutely everything out of the water. If they can already create -basic- Python…

That sounds amazing and also the jailbreak of it via adversarial voice prompting sounds like a horrific vulnerability.

Re: Ferret: A Multimodal Large Language Model

#172

Post by MBCook » Sat, Dec 23, 2023, 8:09 PM UTC

tambourine_man wrote: post #70

Gorgor wrote: post #65

Earlier quoted context omitted.

But no one is training these kinds of models on their personal device. You need compute clusters for that. And they will probably run Linux. I'd be surprised if Microsoft trains their large models in anything else than Linux clusters.

Apple used to sell servers. I don’t thing they should settle for “just use Linux” in such and important field.

Why does the OS matter for training models?

Apple would want to train models as fast as they could. Nvidia provides an off the shelf solution they can just buy and use for a very reasonable price and sell on the second hand market.

If they wanted to use their own hardware they would either need more of it, which would cost a lot and divert production from sellable devices; or they would need to make special chips with much bigger neural engines, which would cost even more.

Also Apple uses public clouds for service stuff. They may not even own any hardware and just be renting it from AWS/Azure/GCP for training.

Re: Ferret: A Multimodal Large Language Model

#173

Post by bugglebeetle » Sat, Dec 23, 2023, 8:11 PM UTC

CaptainOfCoit wrote: post #138

nexuist wrote: post #96

Earlier quoted context omitted.

Siri is really quite dumb. I am confident that a 7B model would be able to provide better responses in over 90% of user queries. I can't even get Siri to reliably set a timer.

Yes, Siri is really dumb. But so is every 7B/13B model out there too.

Depends on if they implement some form of function calling, really. If something like a 7B Mistral fine-tune had access to search and various applications, I imagine it would perform fine and better than Siri.

Re: Ferret: A Multimodal Large Language Model

#174

Post by MBCook » Sat, Dec 23, 2023, 8:13 PM UTC

ayewo wrote: post #95

smoldesu wrote: post #71

Earlier quoted context omitted.

I don't get it, does Apple also make fridges now?

They are implying that even though Apple is a wealthy consumer hardware company that has major spats with nVidia and Samsung, it doesn't always make economic sense to make tools they might need in-house when they can simply buy them from a rival. So rather than invest engineering resources to re-imagine the fridge, they can simply buy them from established manufacturers that make household appliances like Samsung, So…

Not only that, don’t they buy display panels and maybe even storage or RAM chips from Samsung?

Once two giant companies are dealing with each other it can get really complicated to cut everything off.

Re: Ferret: A Multimodal Large Language Model

#175

Post by [deleted] » Sat, Dec 23, 2023, 8:16 PM UTC

n2d4 wrote: post #134

bbor wrote: post #52

Earlier quoted context omitted.

Ok our options Multimodal large language model Large multimodal language model Large language multimodal model Large language model (multimodal) I prefer 1, because this is a multimodal type of an existing technique already referred to as LLM. If I was king, I’d do Omnimodal Linguistic Minds, but no one asks me such things, thank god

I mean, if we want to be silly, what about "Language model (large, multimodal"? :)

[deleted]

Re: Ferret: A Multimodal Large Language Model

#176

Post by tambourine_man » Sat, Dec 23, 2023, 8:16 PM UTC

blackoil wrote: post #105

tambourine_man wrote: post #66

Earlier quoted context omitted.

They use many clouds. But LLM should be their core business and they usually don’t outsource that.

That's not how s/w or any development work. Even if there is a team working on M4 SuperUltra++ which competes with H100, in meanwhile, s/w team will continue to use what is available which may be Intel/AMD PCs with Nvidia GPUs or GoogleCloud/Azure/AWS.

Sure, but there’s also the possibility that they aren’t developing the SuperUltra++

Re: Ferret: A Multimodal Large Language Model

#177

Post by MBCook » Sat, Dec 23, 2023, 8:16 PM UTC

cryogenicfire wrote: post #89

smoldesu wrote: post #44

> FERRET is trained on 8 A100 GPUs with 80GB memory. Huh, even Apple isn't capable of escaping the CUDA trap. Funny to see them go from moral enemies with Nvidia to partially-dependent on them...

I feel like Apple is only testing the waters with AI right now, but perhaps if they get involved enough they'll spend money on their own compute infrastructure? Nvidia is kind of the king at GPU compute right now, and developing comparable hardware is no small or cheap task, but I think Apple is in a very good position to be able to make it work---if they decide to invest in it. But honestly, as far as corporate feud…

> But honestly, as far as corporate feud goes, I feel like companies will happily suck it up if it makes some process cheaper and/or easier

That’s what I think is going on. Apple hated being on the hook for Nvidia’s terrible drivers and chipset/heat problems that ended up causing a ton of warranty repairs.

In this case they’re not a partner, they’re just a normal customer like everyone else. And if Intel comes out with a better AI training card tomorrow Apple can switch over without any worry.

They’re not at the mercy of Nvidia like they were with graphics chips. They’re just choosing (what I assume to be) the best off the shelf hardware for what they need.

Re: Ferret: A Multimodal Large Language Model

#178

Post by xerxes249 » Sat, Dec 23, 2023, 8:18 PM UTC

antiframe wrote: post #159

darthrupert wrote: post #146

Earlier quoted context omitted.

My expectation of Apple is that they lurk in the shadows, looking at what others do while perfecting their own thing. Then on the day of release, they'll be a decade ahead of competition. They've done this a dozen times already.

Which dozen times has Apple released something decades ahead of the competition? I'm blanking on 4-12.

64-bit phones is the easy one.

Re: Ferret: A Multimodal Large Language Model

#179

Post by MBCook » Sat, Dec 23, 2023, 8:19 PM UTC

devinprater wrote: post #149

CaptainOfCoit wrote: post #8

Maybe the abstract of the paper is a better introduction to what this is: > We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates d…

This is going to be great for accessibility! Imagine being blind and loading up a video game and using this to figure out what's around, having everything described locally. I mean, um, well that's what I'd use it for anyway. But knowing Apple, we won't be able to prompt the LLM directly so that probably won't happen until 5 years from now.

The Magnifier app on iOS can already describe whatever you point your phone at in iOS 17.

It’s not going to know an orc from a health potion, but they’re certainly working on the idea in the everyday stuff domain.

Re: Ferret: A Multimodal Large Language Model

#180

Post by MBCook » Sat, Dec 23, 2023, 8:21 PM UTC

cyanydeez wrote: post #160

echelon wrote: post #49

Earlier quoted context omitted.

Boo. But what do we expect from these giants? They're not going to create fertile ground for new competition. The only businesses they foster are those living under thumb and paying tax. I guess I at least hoped for "commoditize the compliments" here. Make Google and OpenAI broadly less special.

it's more likely it's all "stolen" and this is CYA

I seriously doubt that. I’m sure Apple got the rights to whatever they need, it’s not like they’re short on money.

But the fact that they licensed it doesn’t mean that license can be transferred to other people. So it may be that they can only release it for research under the terms of the licenses they got.