Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

apps.apple.com

Post reply on HN
Print view View original

81–90 of 208 posts

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#81

Post by coder543 »

jrvarela56 wrote: post #48
simonw wrote: post #2

Does it save all conversations and let me revisit them later? I use MLC Chat to run Mistral 7B on my iPhone at the moment, but the lack of conversation history is a real nuisance: https://apps.apple.com/us/app/mlc-chat/id6448482937

In your experience, how could these local LLMs become snappier than using streamed API calls? How far are they if not? How soon do you guess they’ll get there? I understand the motivation includes factors other than performance, I’m just curious about performance as it applies to UX.

What does snappier even mean in this context? The latency from connecting to a server over most network connections isn’t really noticeable when talking about text generation. If the server with a beefy datacenter-class GPU were running the same Mistral you can run on your phone, it would be spitting out hundreds of tokens per second. Most responses would appear on your screen before you blink.

There is no expectation that phones will ever be comparable in performance for LLMs.

Mistral runs at a decent clip on phones, but we’re talking like 11 tokens per second, not hundreds of tokens per second.

Server-based models tend to be only slightly faster than Mistral on my phone because they’re usually running much larger, much more accurate/useful models. Models which currently can’t fit onto phones.

Running models locally is not motivated by performance, except if you’re in places without reliable internet.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#82

Post by Our_Benefactors »

pizza wrote: post #67
Zetobal wrote: post #59

Earlier quoted context omitted.

The base models of stablediffusion were always deterministic if you use a deterministic noise scheduler...

I think they’re referring to CUDA (and possibly other similar runtimes) being able to schedule floating point ops non-deterministically, combined with floating point arithmetic being potentially non-associative. I’m not personally sure how big an issue that would be for the output though.

Differences in output are generally varying levels of difficulty of “spot the difference” and rarely changes the overall image composition by much. I always use nondeterministic algos and it doesn’t have any affect on my ability to refine prompts effectively.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#83

Post by halJordan »

sp332 wrote: post #78
furyofantares wrote: post #77

edit: my bad, I misread the price and it's really hard to see the price after you bought it to double check. $10 for something that (I think) doesn't work on most phones but isn't gated to ones it works on feels hostile. Probably there's no way to gate, in that case I'd suggest not charging for it. Or I guess adding a daily usage limit that's lifted with an IAP. I'll admit I was off-put by the price to begin with, wh…

It's $1.99 and the description says: The app requires a Pro iPhone with a minimum of 6GB of RAM. Only the following devices meet the requirement: - iPhone 15 Pro, iPhone 14 Pro, iPhone 13 Pro, iPhone 12 Pro. - iPads: Please check. RAM varies based on model and year.

So if they know it wont work, and do not put that info into the store's compatibility matrix then it's still a bait/switch to me. Compare to the Resident Evil page which does set the store limits on what devices can dl it.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#84

Post by halJordan »

jameshart wrote: post #66
alekseiprokopev wrote: post #35

Here is how to do that on Android: https://github.com/ggerganov/llama.cpp/#android

I don't think running raw llama.cpp under termux in a shell on your phone, after downloading and compiling it from scratch,, is really comparable to 'I made an app'.

[flagged]

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#85

Post by coder543 »

brittlewis12 wrote: post #22
simonw wrote: post #2

Does it save all conversations and let me revisit them later? I use MLC Chat to run Mistral 7B on my iPhone at the moment, but the lack of conversation history is a real nuisance: https://apps.apple.com/us/app/mlc-chat/id6448482937

you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg

EDIT: Attempting to converse with any Q4_K_M 7B parameter model on a 15 Pro Max... the phone just melts down. It feels like it is producing about one token per minute. MLC-Chat can handle 7B parameter models just fine even on a 14 Pro Max, which has less RAM, so I think there is an issue here.

EDIT 2: Even using StableLM, I am experiencing a total crash of the app fairly consistently if I chat in one conversation, then start a new conversation and try to chat in that. On a related note, since chat history is saved... I don't think it's necessary to have a confirmation prompt if the user clicks the "new chat" shortcut in the top right of a chat.

-----

That does seem much nicer than MLC Chat. I really like the selection of models and saving of conversations.

It looks like you’re still using the old version of TinyLlama. The 1.0 release is out now: https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGU...

Microsoft recently re-licensed Phi-2 to be MIT instead of non-commercial, so I would love to see that in the list of models. Similarly, there is a Dolphin-Phi fine tune.

The topic of discussion here is Mistral-7B v0.2, which is also missing from the model list, unfortunately. There are a few Mistral fine tunes in the list, but obviously not the same thing.

I also wish I could enable performance metrics to see how many tokens/sec the model was running at after each message, and to see how much RAM is being used.

On the whole, this app seems really nice!

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#86

Post by amluto »

dilawar wrote: post #43
tionis wrote: post #36

Earlier quoted context omitted.

Yes, you can set the temperature to 0, then they should be deterministic.

Someone mentions temperature in the context of algorithms, can't stop thinking, cool, simulated annealing. Haven't seen temperature used in any other family of algo before this.

If you squint, it’s the same thing. Simulated annealing generally attempts to sample from the Boltzmann distribution. (Presumably because actual annealing is a thermodynamic thing, and you can often think of annealing in a way that the system is a sample from the Boltzmann distribution.)

And softmax is exactly the function that maps energies into the corresponding normalized probabilities under the Boltzmann distribution. And transformers are generally treated as modeling the probabilities of strings, and those probabilities are expressed as energies under the Boltzmann distribution (i.e., logits are on a log scale), and asking your favorite model a question works by sampling from the Boltzmann distribution based on the energies (log probabilities) the model predicts, and you can sample that distribution at any temperature you like.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#87

Post by wahnfrieden »

ricktdotorg wrote: post #70
wahnfrieden wrote: post #4

Earlier quoted context omitted.

I made a free / mostly open source one for iOS that lets you edit the system prompt https://chatonmac.com

trying this out! BTW and FYI i need to reduce the font size on my iOS device to be smaller than i like in order to use your add/replace API key key pages. if the font is "larger than normal" i can't see/focus on the box to enter or paste in the API key. just increase your iOS system font size to trigger this. thanks in advance for fixing, will try out the app!

Thanks for the detailed report - will fix asap, along with releasing the macOS v1.0. I've just soft launched this so far but have more to come so please let me know anything else.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#88

Post by etaioinshrdlu »

Dig1t wrote: post #76
etaioinshrdlu wrote: post #65

Does Apple enforce strict safety and content rules on these types of apps?

What does that mean exactly? Like your phone won’t print text that says something offensive?

I'm concerned Apple won't approve your app for the App store if your chat app says something offensive.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#89

Post by valianteffort »

alchemist1e9 wrote: post #80
Dig1t wrote: post #76

Earlier quoted context omitted.

What does that mean exactly? Like your phone won’t print text that says something offensive?

[flagged]

What the hell are you on about? Apple has rules and guidelines on (user) generated content, GP was asking whether it applied here.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#90

Post by j45 »

geuis wrote: post #27

I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?

An external thunderbolt gpu should work with an Intel MacBook Pro
Post reply on HN