Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ferret: A Multimodal Large Language Model

github.com

Post reply on HN
Print view View original

271–280 of 332 posts

Re: Ferret: A Multimodal Large Language Model

#271

Post by FLT8 »

Booourns wrote: post #235
fbdab103 wrote: post #233

Earlier quoted context omitted.

I would challenge the keyboard autocomplete. I find the Apple suggestions to be frustratingly poor vs my experience on Android.

Out of curiosity have you experienced their autocorrect on iOS 17 because that’s when they updated to be LLM based?

I don't recall exactly when it started happening, but I've been having lots of issues with recent iOS versions rewriting not the last word entered, but the word before that. For example, if I start entering "I went to", it'll sometimes correct to "I want to", but it'll do that after I've typed the "to". I've found lots of similar examples. The retrospective nature of the edits mean I miss a lot of them and makes me appear a lot less literate than I am.

Re: Ferret: A Multimodal Large Language Model

#273

Post by Aerbil313 »

hansoolo wrote: post #253
shrimpx wrote: post #227

Earlier quoted context omitted.

- FaceID - Facial recognition in Photos - "Memories" in Photos - iOS keyboard autocomplete using LLMs. I am bilingual and noticed in the latest iOS it now does multi-language autocomplete and you no longer have to manually switch languages. - Event detection for Calendar - Depth Fusion in the iOS camera app, using ML to take crisper photos - Probably others... The crazy thing is most/all of these run on the device.

To be honest I distrust Microsoft with swift key, but the it recognizes the change in language just smooth. I could switch languages in one sentence and it would understand what I am writing just fine, no Sill suggestions

Apple recommending wrong words when you write in mixed-language was the case in iOS 15, so much I always needed to manually change my keyboard language. But it’s no more in iOS 17. As an example I just typed this entire comment in Turkish keyboard with English autocorrect and suggestions working.

Maybe the (most likely) AI-based thing requires some training though. I got my new iPhone a month or so ago.

Re: Ferret: A Multimodal Large Language Model

#274

Post by Aerbil313 »

theferalrobot wrote: post #258
hosh wrote: post #246

Earlier quoted context omitted.

Apple doesn’t have to win developer trust or build an AI platform. They just have to build a compelling consumer product that can only function with AI, and they are better equipped to do that than Google or Microsoft. It remains to be seen if OpenAI will go that route instead of a business built on training and providing access to foundational models.

>They just have to build a compelling consumer product that can only function with AI Yeah and I'm not talking exclusively about developer trust. Given Apple's current consumer lineup (see Siri, Apple photos, predictive text etc)... we only have evidence that they suck at ML. What makes you think they are going to suddenly transform overnight?

> see Siri, Apple photos, predictive text etc)... we only have evidence that they suck at ML

…or that they only deal with mature tech and not the shiny new thing. Makes sense to me. I don’t doubt everyone will have a personal LLM-based assistant in their phones soon, but with the current rate of improvements to LLMs and AI in general, I’d wait for at least a year more while doing R&D in-house if I were Apple.

Re: Ferret: A Multimodal Large Language Model

#275

Post by whalesalad »

smoldesu wrote: post #44

> FERRET is trained on 8 A100 GPUs with 80GB memory. Huh, even Apple isn't capable of escaping the CUDA trap. Funny to see them go from moral enemies with Nvidia to partially-dependent on them...

Apple silicon is good but it’s designed for a portable. Even the studio and Mac Pro are just laptop chips stitched together. They gotta use heavy duty gear to do heavy duty shit. I know they have a soured relationship with nvidia tho so I would like to see them bolster the AMD/rocm ecosystem. Chances are they’re working on their own stuff here too, though. They are sitting on billions of dollars of liquid cash so I’d imagine they’re using that for some serious R&D.

Re: Ferret: A Multimodal Large Language Model

#276

Post by agentcoops »

hosh wrote: post #246
theferalrobot wrote: post #239

Earlier quoted context omitted.

Given Apple's track record on anything AI related and the terrible state they keep CoreML that not only seems extraordinarily unlikely, it would take a lot of time to win developer trust and that I just don't see happening.

Apple doesn’t have to win developer trust or build an AI platform. They just have to build a compelling consumer product that can only function with AI, and they are better equipped to do that than Google or Microsoft. It remains to be seen if OpenAI will go that route instead of a business built on training and providing access to foundational models.

Yes, this is the most important point and I think somehow least present in even discussions here: the technical question of who produces the best/cheapest LLM/future architecture is considerably less important than who, if anyone, creates a fundamentally new and dominant consumer experience built on AI. Most of the existing players (Google, Meta) would of course prefer that nobody produces such a newly dominant paradigm of computation for end-users since it would greatly reduce their relevance and subsequently revenues. Right now, ChatGPT is the only real contender in this space. However, I think you’re correct that it’s actually Apple who is most likely to be the next who attempts such a paradigm shift. Far too early to bet, but let’s say I wouldn’t be surprised if in five years we end up in a world in which Apple has the consumer monopoly and Microsoft the business monopoly, with Google and Meta falling into irrelevance.

Re: Ferret: A Multimodal Large Language Model

#277

Post by dilippkumar »

bbor wrote: post #52
pests wrote: post #16

Earlier quoted context omitted.

why not LLMM?

Ok our options Multimodal large language model Large multimodal language model Large language multimodal model Large language model (multimodal) I prefer 1, because this is a multimodal type of an existing technique already referred to as LLM. If I was king, I’d do Omnimodal Linguistic Minds, but no one asks me such things, thank god

Some of the modalities in multimodal are non-linguistic. For example, image or video input. In those cases, is it still a language model?

Re: Ferret: A Multimodal Large Language Model

#278

Post by reissbaker »

SpaceManNabs wrote: post #108
reissbaker wrote: post #60

Earlier quoted context omitted.

The paper was definitely cool but doesn't allow you to run particularly large LLMs on iPhones. It allows you to run a certain kind of LLM (sparse ReLU based LLMs) whose weights are somewhere less than 2x RAM. So, 7b Falcon works, but the competitive-with-gpt-3.5-turbo LLMs are still out of reach (and aren't ReLU based, although maybe that could change in the future). And nothing is competitive with GPT-4 right now. O…

> And nothing is competitive with GPT-4 right now. You mean nothing available? Or you mean nothing that public knows exists? The answers to those two questions are different. There are definitely products that aren't available but the public knows exist and are upcoming that are in GPT-4's ballpark.

I mean nothing that is able to be benchmarked and validated by third parties is GPT-4 quality. I know there are upcoming releases that are hyped as being equal to GPT-4, e.g. Gemini Ultra, which I am very excited to get my hands on — but regardless, Ultra is not small enough to run on phones, even using the sparse ReLU flash memory optimization. And we'll see how it benchmarks once it's released; according to some benchmarks Gemini Pro has somewhat underperformed GPT-3.5-Turbo [1], despite Google's initial claims. (Although there are criticisms of that benchmarking, and it does beat the current 1106 version of GPT-3.5-Turbo on the Chatbot Arena leaderboard [2], although it slightly underperforms the previous 0613 version.)

1: https://arxiv.org/pdf/2312.11444.pdf

2: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Re: Ferret: A Multimodal Large Language Model

#279

Post by dwaite »

shrimpx wrote: post #216

Apple has been looking sleepy on LLMs, but they've been consistently evolving their hardware+software AI stack, without much glitzy advertising. I think they could blow away Microsoft/OpenAI and Google, if suddenly a new iOS release makes the OpenAI/Bard chatbox look laughably antiquated. They're also a threat to Nvidia, if a significant swath of AI usage switches over to Apple hardware. Arm and TSMC would stand to w…

> Apple has been looking sleepy on LLMs, but they've been consistently evolving their hardware+software AI stack, without much glitzy advertising

They don't sell compute time to other companies to run AI, or massive custom hardware for AI training.

They aren't after VC funding.

Their core business isn't threatened by AI being "the evolution of search"

Product-wise, so far all you hear is messaging around things like pointing out the applicability of the M3 Max for running ML models.

Until they have real consumer products ready, they only need to keep tabs on analysts, with lip service at financial meetings.

Re: Ferret: A Multimodal Large Language Model

#280

Post by sagz »

devinprater wrote: post #142

They're already going multi-modal? Holy crap, if google can't deliver in the accessibility space for this (image descriptions better than "the logo for the company"), then I'll definitely go back to Apple. I mean I do hope Apple cleans out bugs and makes VoiceOver feel like it won't fall over if I breathed hard, but their image descriptions, even without an LLM, are already clean and clear. More like "A green logo on…

Google's Lookout app (accessibility for the blind and visually impaired) was updated ~6 months ago with a multimodal LLM already.

It uses the Flamingo model family: https://deepmind.google/discover/blog/tackling-multiple-task...

Post reply on HN