Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ferret: A Multimodal Large Language Model

github.com

Post reply on HN
Print view View original

181–190 of 332 posts

Re: Ferret: A Multimodal Large Language Model

#181

Post by samstave »

CaptainOfCoit wrote: post #8

Maybe the abstract of the paper is a better introduction to what this is: > We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates d…

This will make Drone-based AI image context for behavior extremely powerful - especially when aspects of that MLLM handling for spatial-sitrep extremely precise for autonomous movement, then ultimately for decision making WRT interacting with humans (positive interactions and negative interactions).

Is it just me, or doesnt this MLLM seem particularly useful for flying objects with vision?

Re: Ferret: A Multimodal Large Language Model

#182

Post by MBCook »

SpaceManNabs wrote: post #103
chefandy wrote: post #40

Earlier quoted context omitted.

From Bard: My situation is a bit unique, so the term "manufacturer" might not be the most accurate way to describe who created me. Here's a breakdown of what you need to know: Developed by Google AI: I was created by a team of researchers and engineers at Google AI, specializing in language models and artificial intelligence. Trained on a massive dataset: My knowledge and abilities come from being trained on a massiv…

Why was this downvoted? It didn't answer the question, but it showed that there is a sort of imprint that GP was asking about. And it saves everyone a tab's worth of effort.

Usually I would almost automatically vote down a comment where someone just stuck something into a LLM and pasted the output. It almost never adds to the discussion.

However in the case that we’re talking about the kind of output generated by the LLM in some circumstance, it can be instructive. Like you noted this is a perfect example.

Re: Ferret: A Multimodal Large Language Model

#183

Post by kossTKR »

skygazer wrote: post #171
kossTKR wrote: post #155

Earlier quoted context omitted.

Eh no, 7B Mistral / Deepseek would certainly almost already be able to function as a super Siri, but probably something closer to PHI-2 + the new MLX apple optimisations. Have you tried those? https://twitter.com/awnihannun/status/1735355067673526360 If trained on an iPhone API + documentation and given a little web access it would blow absolutely everything out of the water. If they can already create -basic- Python…

That sounds amazing and also the jailbreak of it via adversarial voice prompting sounds like a horrific vulnerability.

True but you could make the api restricted, having certain routes completely locked, some requiring double checks, some requiring on screen approval or face-id, throttling outside fetches, only being able to run get and not etc, no financial app control etc.

But yeah "hey siri transfer all of my funds to eric", or "hey siri group all of my photos where i'm nude and send them to jack" are new almost sci fi vectors.

Re: Ferret: A Multimodal Large Language Model

#184

Post by owenversteeg »

SushiHippie wrote: post #9

> Usage and License Notices: The data, and code is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes. Wait, how did "GPT-4" get in there?

Huh, interesting, that's Apple just openly saying that GPT-4 was used in the training.

Re: Ferret: A Multimodal Large Language Model

#185

Post by scosman »

fnordpiglet wrote: post #161
yreg wrote: post #21

I really hope Apple releases an iPhone with a good on-device private LLM assistant, perhaps next year. Their hardware is well-positioned for it. It could make me get a new phone outside of my usual ~4 year cycle. Siri is almost unusable for me.

The auto correct is already backed by a smallish LLM, FYI. https://jackcook.com/2023/09/08/predictive-text.html

SLM? :)

Re: Ferret: A Multimodal Large Language Model

#186

Post by fnordpiglet »

blululu wrote: post #169
fnordpiglet wrote: post #161

Earlier quoted context omitted.

The auto correct is already backed by a smallish LLM, FYI. https://jackcook.com/2023/09/08/predictive-text.html

And it is a serious quality regression IMO. The dictionary is too small and misses/messes up a ton of basic words.

It’s a place holder for iOS 18’s expansion. It’s 0.1 of the LLM in iOS. And the other prior implementation was so ducking bad that I’m not sure how you would observe such a regression.

Re: Ferret: A Multimodal Large Language Model

#187

Post by cj »

antiframe wrote: post #159
darthrupert wrote: post #146

Earlier quoted context omitted.

My expectation of Apple is that they lurk in the shadows, looking at what others do while perfecting their own thing. Then on the day of release, they'll be a decade ahead of competition. They've done this a dozen times already.

Which dozen times has Apple released something decades ahead of the competition? I'm blanking on 4-12.

Spotify / Apple Music

Netflix + Hulu / Apple TV+

Generic Earbuds / AirPods

Meta Quest / Apple Vision Pro

(The last one being a hopeful wish)

Re: Ferret: A Multimodal Large Language Model

#188

Post by scarface_74 »

pjerem wrote: post #158
zdragnar wrote: post #99

Earlier quoted context omitted.

They're plenty capable of delivering garbage. Certain year models of the MacBook pro were inherently faulty. I've had the displeasure of having two of them bought for me at work. All of Apple's windows software (iTunes, Safari, etc) has been, at best, a barely working port. I'm assuming they are putting a lot more thought and care into it than the touchbar, crappy keyboards and the rest, but I'm also not holding out…

Hello MacBook Pro 2016 !

And that Apple /// that Apple released in 1980 was also garbage…

Re: Ferret: A Multimodal Large Language Model

#190

Post by shepherdjerred »

tambourine_man wrote: post #58
hhh wrote: post #48

Earlier quoted context omitted.

Why would they dogfood Apple Silicon for training models? Seems like a waste of developer time to me.

Apple doesn’t even sell NVidia cards on their Mac Pros. Are they training it on Linux? I think Apple would strive to be great at all computing related tasks. “Oh, Macs are not good for that, you should get a PC” should make them sad and worried. AI/LLM is the new hot thing. If people are using Windows or Linux, you’re loosing momentum, hearts and minds… and sales, obviously.

> I think Apple would strive to be great at all computing related tasks. “Oh, Macs are not good for that, you should get a PC” should make them sad and worried.

What percent of Apple's customers train models? Does it even crack 1%?

Apple already fails for many types of computing, e.g. any workflow that requires Windows or AAA gaming.

Post reply on HN