Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

apps.apple.com

Post reply on HN
Print view View original

191–200 of 208 posts

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#191

Post by ngai_aku »

wahnfrieden wrote: post #182
ngai_aku wrote: post #179

Earlier quoted context omitted.

I was excited when I saw this, but I'm having trouble with it (and it looks like I'm not the only one). As others have pointed out, the download link on your site does open TestFlight. I've since deleted that version and installed the official version from the AppStore after revisiting this thread in search of answers. I now have the full version installed on my iPhone 15 pro, and I have added my OpenAI key, but none…

Apologies for the rough edges and bad experience - I’ve just soft launched without announcement til this post. I will have a hotfix up soon. Thanks for the report.

No stress. Best of luck!

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#192

Post by urbandw311er »

sockaddr wrote: post #94
brittlewis12 wrote: post #18

Earlier quoted context omitted.

would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg

Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.

Exactly the same here - full lock up for 2 minutes without being able to reboot even with hardware buttons.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#193

Post by urbandw311er »

Are there any models out there that don’t come trained or tweaked or system prompted into somebody else’s idea of ethical or professional conduct? I tested out a bunch of these apps and asked them to write an explicit story to see if they would, and despite this being entirely legal, none would do so. Are we entering some new Orwellian era?

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#194

Post by brittlewis12 »

coder543 wrote: post #100
brittlewis12 wrote: post #99

Earlier quoted context omitted.

Wow, thanks so much for taking the time to test it out and share such great feedback! Thrilled about all those developments! More model options as well as link-based GGUF downloads on the way. On the 7b models: I’m very sorry for the poor experience. I wouldn’t recommend 7b over Q2_K at the moment, unless you’re on a 16GB iPad (or an Apple Silicon Mac!). This needs to be much clearer, as you observed the consequences…

I just checked and MLC Chat is running the 3-bit quantized version of Mistral-7B. It works fine on the 14 Pro Max (6GB RAM) without crashing, and is able to stay resident in memory on the 15 Pro Max (8GB RAM) when switching with another not-too-heavy app. 2-bit quantization just feels like a step too far, but I’ll give it a try. Regarding credit, I definitely don’t need any. Just happy to see someone working on a bet…

FYI, just submitted a new update for review with a few small but hopefully noticeable changes, thanks in no small part to your feedback:

1. StableLM Zephyr 3b Q4_K_M is now the built-in model, replacing the Q6_K variant.

2. More aggressive RAM headroom calculation, with forced fallback to CPU rather than failing to load or crashing.

3. New status indicator for Metal when model is loaded (filled bolt for enabled, vs slashed bolt for disabled.)

4. Metal will now also be enabled for devices with 4GB RAM or less, but only when the selected model can comfortably fit in RAM. Previously, only devices with at least 6GB had Metal enabled.

Thank you so much again for your time!

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#195

Post by brittlewis12 »

johngalt2600 wrote: post #61

Where to leave feedback? I am trying the Mistral dolphin model but getting GGML ASSERT errors referencing Users/tito lol (not me). Using iPhone 14 Pro Max.

Sorry for the confusing experience, and thank you for sharing this!

I’ve just submitted a new update for review with a few small but hopefully noticeable changes, thanks in no small part to your feedback:

1. StableLM Zephyr 3b Q4_K_M is now the built-in model, replacing the Q6_K variant.

2. More aggressive RAM headroom calculation, with forced fallback to CPU rather than failing to load as you observed, or crashing outright in some nasty edge cases.

3. New status indicator for Metal when model is loaded (filled bolt for enabled, vs slashed bolt for disabled.)

4. Metal will now also be enabled for devices with 4GB RAM or less, but only when the selected model can comfortably fit in RAM. Previously, only devices with at least 6GB had Metal enabled.

Thank you so much for taking the time to test and share your experience! Feel free to reach out anytime at britt [at] bl3 [dot] dev.

Britt

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#196

Post by brittlewis12 »

sockaddr wrote: post #94
brittlewis12 wrote: post #18

Earlier quoted context omitted.

would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg

Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.

Quick follow-up:

I’ve just submitted a new update for review with a few small but hopefully noticeable changes, thanks to your feedback:

1. StableLM Zephyr 3b Q4_K_M is now the built-in model, replacing the Q6_K variant.

2. More aggressive RAM headroom calculation, with forced fallback to CPU rather than failing to load or crashing/hanging in such a nasty fashion.

3. New status indicator for Metal when model is loaded (filled bolt for enabled, vs slashed bolt for disabled.)

4. Metal will now also be enabled for devices with 4GB RAM or less, but only when the selected model can comfortably fit in RAM. Previously, only devices with at least 6GB ever had Metal enabled.

I really appreciate your taking the time to test — the hanging you experienced was unacceptable, and I truly am sorry for the inconvenience. I hope you’ll give it another chance once this update is live, but either way I’m grateful for your help in isolating and eliminating this issue!

Britt

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#197

Post by nl »

sp332 wrote: post #187
nl wrote: post #185

Earlier quoted context omitted.

Yes, but this whole block of text gets passed to the LLM on each call as the conversation history. The [INST] tags tell the LLM which parts were inputs (system instructions) as opposed to outputs.

Well, that's not what a system prompt is. There is a difference between system prompts and regular conversational inputs. https://platform.openai.com/docs/guides/prompt-engineering/t...

In actual fact system prompts are just the instruction tuning prompts that provide instructions formatted in the correct way (ie, with [INST]). It's just text all the way down.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#198

Post by brittlewis12 »

urbandw311er wrote: post #192
sockaddr wrote: post #94

Earlier quoted context omitted.

Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.

Exactly the same here - full lock up for 2 minutes without being able to reboot even with hardware buttons.

I’m very sorry to hear you had such a poor experience as well. I’m sure it’s little consolation at this point having been inconvenienced as you have — it’s certainly not what I aim for in my work!

I’ve just submitted a new update for review with a number of small but material changes to address these issues: https://news.ycombinator.com/item?id=38920916

I hope you’ll consider giving it another shot once that’s live, and thank you for taking the time not just to test but also to report your experience!

Britt

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#199

Post by swyx »

brittlewis12 wrote: post #163
swyx wrote: post #122

Earlier quoted context omitted.

great high effort answer, thanks so much! to prod you to sell yourself a bit more - what is the goal/selling point of cnvrs?

Oh man I’m a big fan, swyx!! Latent Space & AI.engineer are fantastic resources to the community. Thank you for the kind words & the prompt! It’s still early days, but at a high level, I have a few goals: - expand accessibility and increase awareness of the power & viability of small models — the scene can be quite impenetrable for many! - provide the an easy to use, attractive, efficient app that’s a good platform c…

i dont dream of truly private ai like that haha. im a pretty open book. but very very glad to see more options in the local ai space!

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#200

Post by kkielhofner »

jorvi wrote: post #188
kkielhofner wrote: post #169

Earlier quoted context omitted.

Snapchat filters, iPhone photo processing/speech to text/always-on Hey Siri/OCR/object detection and segmentation - there are countless applications and functionality doing this on device today (and for years). For something like the RAG approach I mentioned the sync and coordination of your local content to a remote API would be more taxing on the battery just in terms of the radio than what we already see from on d…

> Snapchat filters, iPhone photo processing/speech to text/always-on Hey Siri/OCR/object detection and segmentation - there are countless applications and functionality doing this on device today (and for years) But those are peanuts compared to what will be possible in the (near) future. You think content-aware fill is neat? Wait until you can zoom out of a photo 50% or completely change the angle. That’ll costs gob…

> That’ll costs gobs of processing power and thus time and battery

The exact same thing was said about the functionality we're describing yet there it is. Imagine describing that to someone in 2010 who's already complaining about iPhone battery life. The response would be carbon-copy to yours.

In five years from the iPhone 8 to the iPhone 14 TOPS on the neural engine went from 0.6 to 17[0]. The iPhone 15 more than doubled that and stands at 35 TOPS[1]. Battery life is better than ever and that's a 58x gain just in neural, not even GPU, CPU, performance cores, etc.

Over that same period of time Nvidia GPUs only increased about 9x[2] - they're pushing the fundamentals much harder as a law of large numbers-ish issue.

So yeah, I won't have to wait long for zoom out of a photo 50%, completely change the angle, or who knows what else to be done locally. In fact, for these use cases increasingly advanced optics, processing, outside visual range sensors, etc, etc makes my point even more - even more data going to the cloud when the device is best suited to be doing it anyway.

Look at it this way - Apple sold over 97 million iPhones in 2023. Assuming the lower averages that's 1,649,000,000 combined TOPS out there.

Cloud providers benefit from optimization and inherent oversubscription but by comparison Nvidia sold somewhere around 500,000,000 TFLOPS worth of H100s last year.

Mainframe and serial terminal to desktop to thin client and terminal server - around and around we go.

[0] - https://appleinsider.com/articles/22/09/26/how-iphone-speeds...

[1] - https://www.counterpointresearch.com/insights/iphone-15-usb-...

[2] - https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...

Post reply on HN