Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

apps.apple.com

Post reply on HN
Print view View original

41–50 of 208 posts

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#41

Post by ionwake »

wahnfrieden wrote: post #4
hananova wrote: post #3

Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will respond to later questions.

I made a free / mostly open source one for iOS that lets you edit the system prompt https://chatonmac.com

Amazing! Does it submit any data online ?

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#42

Post by jey »

geuis wrote: post #27

I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?

I'd suggest using a cloud VM with a GPU attached. For normal stuff like LLM inference, I just rent an instance with a small (cheap) GPU. But when I need to do something more exotic like train an image model from scratch, I can temporarily spin up a cluster that has high-end expensive A100s. This way I don't have to invest in expensive hardware like an M3 that can still only do a small part of the full range.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#43

Post by dilawar »

tionis wrote: post #36
TacticalCoder wrote: post #17

Are these LLMs you can run locally giving answers deterministically just as with, say, StableDiffusion? In StableDiffusion if you reuse the exact same version of SD / model and same query and seed, you always get the same result (at least I think so).

Yes, you can set the temperature to 0, then they should be deterministic.

Someone mentions temperature in the context of algorithms, can't stop thinking, cool, simulated annealing. Haven't seen temperature used in any other family of algo before this.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#44

Post by kiratp »

geuis wrote: post #27

I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?

We should be happy that compute is once again improving and machines are getting outdated rapidly. Which is better - a world where your laptop is competitive for 5+ years but everything stays the same? Or one where entire new realms of advancement open up every 18 months?

It’s a no contest option 2 for me.

Just use llama.cpp with any of the available UIs. It will be usable with 4 but quantization on CPU. You can use any of the “Q4_M” “GGUF” models that TheBloke puts out on Huggingface.

https://github.com/ggerganov/llama.cpp

UI projects in description.

https://huggingface.co/TheBloke

A closed source option is LMStudio.

https://lmstudio.ai/

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#45

Post by TN1ck »

Firmwarrior wrote: post #14

Is it weird if I carry a phone with this and a solar charger around at all times, in case I suddenly get hurled back in time?

Also put the latest dump of Wikipedia on your phone (Kiwix is great for this). The complete English one with images is about 100 GB.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#46

Post by kiratp »

muricula wrote: post #32
geuis wrote: post #27

I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?

Does your mac support an external GPU? A mid to high end nvidia card may or may not outperform the M3 GPU at a lower or similar price. You can also stick it in a PC or resell it separately.

No Nvidia drivers for MacOS.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#48

Post by jrvarela56 »

simonw wrote: post #2

Does it save all conversations and let me revisit them later? I use MLC Chat to run Mistral 7B on my iPhone at the moment, but the lack of conversation history is a real nuisance: https://apps.apple.com/us/app/mlc-chat/id6448482937

In your experience, how could these local LLMs become snappier than using streamed API calls? How far are they if not? How soon do you guess they’ll get there?

I understand the motivation includes factors other than performance, I’m just curious about performance as it applies to UX.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#50

Post by JimDabell »

TacticalCoder wrote: post #17

Are these LLMs you can run locally giving answers deterministically just as with, say, StableDiffusion? In StableDiffusion if you reuse the exact same version of SD / model and same query and seed, you always get the same result (at least I think so).

Even with Stable Diffusion, determinism is “best effort”- there are flags you can set in Torch to make it more deterministic at a performance cost, but it’s explicitly disclaimed:

https://pytorch.org/docs/stable/notes/randomness.html

Post reply on HN