Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

apps.apple.com

Post reply on HN
Print view View original

201–208 of 208 posts

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#201

Post by coder543 »

brittlewis12 wrote: post #194
coder543 wrote: post #100

Earlier quoted context omitted.

I just checked and MLC Chat is running the 3-bit quantized version of Mistral-7B. It works fine on the 14 Pro Max (6GB RAM) without crashing, and is able to stay resident in memory on the 15 Pro Max (8GB RAM) when switching with another not-too-heavy app. 2-bit quantization just feels like a step too far, but I’ll give it a try. Regarding credit, I definitely don’t need any. Just happy to see someone working on a bet…

FYI, just submitted a new update for review with a few small but hopefully noticeable changes, thanks in no small part to your feedback: 1. StableLM Zephyr 3b Q4_K_M is now the built-in model, replacing the Q6_K variant. 2. More aggressive RAM headroom calculation, with forced fallback to CPU rather than failing to load or crashing. 3. New status indicator for Metal when model is loaded (filled bolt for enabled, vs s…

The fallback does seem to work! Although the 4-bit 7B models only run at 1 token every several seconds.

I still wish Phi-2, Dolphin Phi-2, and TinyLlama-Chat-v1.0 were available, but I understand you have plans to make it easier to download any model in the future.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#202

Post by jallbrit »

simonw wrote: post #115
jrvarela56 wrote: post #48

Earlier quoted context omitted.

In your experience, how could these local LLMs become snappier than using streamed API calls? How far are they if not? How soon do you guess they’ll get there? I understand the motivation includes factors other than performance, I’m just curious about performance as it applies to UX.

Honestly I think being able to run any kind of LLM on a phone is a miracle. I'm astonished at how good (and how fast) Mistral 7B runs under MLC Chat on iOS, considering the constraints of the device. I don't use it as more than a cool demo though, because the large hosted LLMs (I tend to mostly use GPT-4) are massively more powerful. But... I'm still intrigued at the idea of a local, slow LLM on my phone enhanced wit…

How do you use GLT-4 frequently with how low the usage cap is?

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#203

Post by urbandw311er »

brittlewis12 wrote: post #198
urbandw311er wrote: post #192

Earlier quoted context omitted.

Exactly the same here - full lock up for 2 minutes without being able to reboot even with hardware buttons.

I’m very sorry to hear you had such a poor experience as well. I’m sure it’s little consolation at this point having been inconvenienced as you have — it’s certainly not what I aim for in my work! I’ve just submitted a new update for review with a number of small but material changes to address these issues: https://news.ycombinator.com/item?id=38920916 I hope you’ll consider giving it another shot once that’s live,…

Thanks. I did test your new version but unfortunately similar issues. App completely hung and entire OS was sluggish. iPhone 13 Pro, iOS 17.1.2. Unfortunately I won’t have time to test any more but very good luck with the project.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#204

Post by madlag »

wannabag wrote: post #144
madlag wrote: post #141

I love the idea, that's the future. However you should be aware that the explanation of second law of thermodynamics generated by the LLM you used in your app store screenshot is wrong: the LLM has it backwards. Energy transfers to less stable states from more stable states, and not the reverse. (I use LLMs for science education apps like https://apps.apple.com/fr/app/explayn-learn-chemistry/id6448... , so I am quite…

Oh, that's an interesting app and in French too... is that something you plan to have on Android as well?

Yes, it's Unity based, so quite easy. There is another version on Quest too, so running on Android : https://www.meta.com/fr-fr/experiences/6113695908674751/ .

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#205

Post by madlag »

Horffupolde wrote: post #142
madlag wrote: post #141

I love the idea, that's the future. However you should be aware that the explanation of second law of thermodynamics generated by the LLM you used in your app store screenshot is wrong: the LLM has it backwards. Energy transfers to less stable states from more stable states, and not the reverse. (I use LLMs for science education apps like https://apps.apple.com/fr/app/explayn-learn-chemistry/id6448... , so I am quite…

How do you define stability in that context?

Stability is actually defined by having a lower energy level. That explains why energy can only flow from a less stable system to a more stable system : the more stable system does not have available energy to give.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#206

Post by bloody-crow »

woadwarrior01 wrote: post #181
bloody-crow wrote: post #180

This might be the best reason to consider a Pro model next time I'm upgrading my iPhone.

iPhone 15 and iPhone 14 Pro, 14 Pro Max have exactly the same CPU and amount of RAM (Apple A16 Bionic and 6GB). This is also true for iPhone 14 and iPhone 13 Pro, Pro Max (Apple A15 Bionic and also 6GB).

I don't play games or do anything too resource-demanding on my phone normally. Pro models typically have more memory than non-pro models and running LLMs on device might be the only scenario where it can realistically make a difference for me.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#207

Post by woadwarrior01 »

bloody-crow wrote: post #206
woadwarrior01 wrote: post #181

Earlier quoted context omitted.

iPhone 15 and iPhone 14 Pro, 14 Pro Max have exactly the same CPU and amount of RAM (Apple A16 Bionic and 6GB). This is also true for iPhone 14 and iPhone 13 Pro, Pro Max (Apple A15 Bionic and also 6GB).

I don't play games or do anything too resource-demanding on my phone normally. Pro models typically have more memory than non-pro models and running LLMs on device might be the only scenario where it can realistically make a difference for me.

Smaller 3B LLMs (like phi-2) work fine on newer non pro models, at full context lengths. Running 7B models on even 8GB iPhone 15 Pro and Pro Max phones involves reducing the context lengths to 1k or fewer tokens, because the full context length KV cache won't fit on these devices.
Post reply on HN