Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

apps.apple.com

Post reply on HN
Print view View original

91–100 of 208 posts

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#91

Post by TN1ck »

peterburkimsher wrote: post #71
TN1ck wrote: post #45

Earlier quoted context omitted.

Also put the latest dump of Wikipedia on your phone (Kiwix is great for this). The complete English one with images is about 100 GB.

And for older phones, Wiki2Touch is a jailbreak app that does the same without images in about 14 GB. I'm writing a Wiki2Touch archive viewer for more modern iOS, because my iPhone SE doesn't have 100 GB free for Kiwix.

Kiwix supports any .zim file, there are plenty of different wikipedia versions available, check https://download.kiwix.org/zim/wikipedia/?C=S;O=D

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#92

Post by dazzaji »

I’m intrigued and currently downloading this app. Love the idea of having offline direct access to this model. One small-ish thing though: Looks like the URL for the privacy policy (http://opusnoma.com/privacy) linked from the App Store page goes nowhere. Actually, opusnoma.com is likewise offline.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#93

Post by peterburkimsher »

TN1ck wrote: post #91
peterburkimsher wrote: post #71

Earlier quoted context omitted.

And for older phones, Wiki2Touch is a jailbreak app that does the same without images in about 14 GB. I'm writing a Wiki2Touch archive viewer for more modern iOS, because my iPhone SE doesn't have 100 GB free for Kiwix.

Kiwix supports any .zim file, there are plenty of different wikipedia versions available, check https://download.kiwix.org/zim/wikipedia/?C=S;O=D

Thanks, but wikipedia_en_all_nopic_2023-12.zim is still 56 GB, whereas the BZ2-compressed Wiki2Touch archives are only about 14 GB for the latest (and only 8 GB for an archive from 2012 which I'm using).

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#94

Post by sockaddr »

brittlewis12 wrote: post #18
hananova wrote: post #3

Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will respond to later questions.

would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg

Do not download this.

I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#95

Post by winstonschen »

etaioinshrdlu wrote: post #65

Does Apple enforce strict safety and content rules on these types of apps?

This app got through review pretty easily, especially since I flagged potentially offensive content which makes it age 12+. In comparison to social media these apps are positively angelic.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#97

Post by [deleted] »

halJordan wrote: post #83
sp332 wrote: post #78

Earlier quoted context omitted.

It's $1.99 and the description says: The app requires a Pro iPhone with a minimum of 6GB of RAM. Only the following devices meet the requirement: - iPhone 15 Pro, iPhone 14 Pro, iPhone 13 Pro, iPhone 12 Pro. - iPads: Please check. RAM varies based on model and year.

So if they know it wont work, and do not put that info into the store's compatibility matrix then it's still a bait/switch to me. Compare to the Resident Evil page which does set the store limits on what devices can dl it.

[deleted]

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#98

Post by winstonschen »

halJordan wrote: post #83
sp332 wrote: post #78

Earlier quoted context omitted.

It's $1.99 and the description says: The app requires a Pro iPhone with a minimum of 6GB of RAM. Only the following devices meet the requirement: - iPhone 15 Pro, iPhone 14 Pro, iPhone 13 Pro, iPhone 12 Pro. - iPads: Please check. RAM varies based on model and year.

So if they know it wont work, and do not put that info into the store's compatibility matrix then it's still a bait/switch to me. Compare to the Resident Evil page which does set the store limits on what devices can dl it.

There is no way to specify iPhone models or memory capacity when submitting an app to App Store. Believe me - I spent several days trying.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#99

Post by brittlewis12 »

coder543 wrote: post #85
brittlewis12 wrote: post #22

Earlier quoted context omitted.

you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg

EDIT: Attempting to converse with any Q4_K_M 7B parameter model on a 15 Pro Max... the phone just melts down. It feels like it is producing about one token per minute. MLC-Chat can handle 7B parameter models just fine even on a 14 Pro Max, which has less RAM, so I think there is an issue here. EDIT 2: Even using StableLM, I am experiencing a total crash of the app fairly consistently if I chat in one conversation, th…

Wow, thanks so much for taking the time to test it out and share such great feedback!

Thrilled about all those developments! More model options as well as link-based GGUF downloads on the way.

On the 7b models: I’m very sorry for the poor experience. I wouldn’t recommend 7b over Q2_K at the moment, unless you’re on a 16GB iPad (or an Apple Silicon Mac!). This needs to be much clearer, as you observed the consequences can be severe. The larger models, and even 3b Q6_K can be crash prone due to memory pressure. Will work on improve handling of low level out-of-memory errors very soon.

Will also investigate the StableLM crashes, I’m sorry about that! Hopefully Testflight recorded a trace. Just speculating, it may be a similar issue to the larger models, due to the higher-fidelity quant (Q6_K) combined with the context length eventually running out of RAM. Could you give the Q4_K_M a shot? I heard something similar from a friend yesterday, I’m curious if you have a better time with that — perhaps that’s a more sensible default.

Re: the overly-protective new chat alert, I agree, thanks for the suggestion. I’ll incorporate that into the next build. Can I credit you? Let me know how you’d like for me to refer to you, and I’d be happy to.

Finally, please feel free to email me any further feedback, and thanks again for your time and consideration!

britt [at] bl3 [dot] dev

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#100

Post by coder543 »

brittlewis12 wrote: post #99
coder543 wrote: post #85

Earlier quoted context omitted.

EDIT: Attempting to converse with any Q4_K_M 7B parameter model on a 15 Pro Max... the phone just melts down. It feels like it is producing about one token per minute. MLC-Chat can handle 7B parameter models just fine even on a 14 Pro Max, which has less RAM, so I think there is an issue here. EDIT 2: Even using StableLM, I am experiencing a total crash of the app fairly consistently if I chat in one conversation, th…

Wow, thanks so much for taking the time to test it out and share such great feedback! Thrilled about all those developments! More model options as well as link-based GGUF downloads on the way. On the 7b models: I’m very sorry for the poor experience. I wouldn’t recommend 7b over Q2_K at the moment, unless you’re on a 16GB iPad (or an Apple Silicon Mac!). This needs to be much clearer, as you observed the consequences…

I just checked and MLC Chat is running the 3-bit quantized version of Mistral-7B. It works fine on the 14 Pro Max (6GB RAM) without crashing, and is able to stay resident in memory on the 15 Pro Max (8GB RAM) when switching with another not-too-heavy app. 2-bit quantization just feels like a step too far, but I’ll give it a try.

Regarding credit, I definitely don’t need any. Just happy to see someone working on a better LLM app!

Post reply on HN