Where to leave feedback? I am trying the Mistral dolphin model but getting GGML ASSERT errors referencing Users/tito lol (not me). Using iPhone 14 Pro Max.
I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
61–70 of 208 posts
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#62Is it weird if I carry a phone with this and a solar charger around at all times, in case I suddenly get hurled back in time?
A machine that tells you that the Golden State Warriors won the 2012 Stanley Cup by bowling a perfect 300 would be invaluable in 1602
In that case no need for a LLM, just a wikipedia dump with a full-text index is enough.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#63Where to leave feedback? I am trying the Mistral dolphin model but getting GGML ASSERT errors referencing Users/tito lol (not me). Using iPhone 14 Pro Max.
Which app are you trying? The app posted here is Offline Chat, which doesn’t have a choice of model and works fine on iPhone 14 Pro.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#64Earlier quoted context omitted.
I made a free / mostly open source one for iOS that lets you edit the system prompt https://chatonmac.com
Amazing! Does it submit any data online ?
No.
I definitely do not want any liability of user-generated content or PII or similar. I have no analytics, besides the standard Apple opt-in crash/reporting (not using any 3rd-party service and not sending anything to my own servers).
It downloads configuration from GitHub and HuggingFace directly. It also has OpenAI integration, directly to their servers via BYOK.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#65Does Apple enforce strict safety and content rules on these types of apps?
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#66Here is how to do that on Android: https://github.com/ggerganov/llama.cpp/#android
I don't think running raw llama.cpp under termux in a shell on your phone, after downloading and compiling it from scratch,, is really comparable to 'I made an app'.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#67Earlier quoted context omitted.
Even with Stable Diffusion, determinism is “best effort”- there are flags you can set in Torch to make it more deterministic at a performance cost, but it’s explicitly disclaimed: https://pytorch.org/docs/stable/notes/randomness.html
The base models of stablediffusion were always deterministic if you use a deterministic noise scheduler...
I think they’re referring to CUDA (and possibly other similar runtimes) being able to schedule floating point ops non-deterministically, combined with floating point arithmetic being potentially non-associative. I’m not personally sure how big an issue that would be for the output though.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#68I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?
We should be happy that compute is once again improving and machines are getting outdated rapidly. Which is better - a world where your laptop is competitive for 5+ years but everything stays the same? Or one where entire new realms of advancement open up every 18 months? It’s a no contest option 2 for me. Just use llama.cpp with any of the available UIs. It will be usable with 4 but quantization on CPU. You can use…
“New realms of advancement” could open up because of faster computation algorithms. Those hypothetical scenarios don’t have to be mutually exclusive.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#69Earlier quoted context omitted.
We should be happy that compute is once again improving and machines are getting outdated rapidly. Which is better - a world where your laptop is competitive for 5+ years but everything stays the same? Or one where entire new realms of advancement open up every 18 months? It’s a no contest option 2 for me. Just use llama.cpp with any of the available UIs. It will be usable with 4 but quantization on CPU. You can use…
i love this perspective! makes me really happy of the advancements going around, and not feel sad about my macbook m1 getting old
[deleted]
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#70Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will respond to later questions.
I made a free / mostly open source one for iOS that lets you edit the system prompt https://chatonmac.com
trying this out!
BTW and FYI i need to reduce the font size on my iOS device to be smaller than i like in order to use your add/replace API key key pages. if the font is "larger than normal" i can't see/focus on the box to enter or paste in the API key. just increase your iOS system font size to trigger this. thanks in advance for fixing, will try out the app!