101–110 of 208 posts
Earlier quoted context omitted.
I don't think running raw llama.cpp under termux in a shell on your phone, after downloading and compiling it from scratch,, is really comparable to 'I made an app'.
[flagged]
More options is a good thing, nobody said a manual process was better than an app.
Earlier quoted context omitted.
Mistral Instruct does use a system prompt. You can see the raw format here: https://www.promptingguide.ai/models/mistral-7b#chat-templat... and you can see how LllamaIndex uses it here (as an example): https://github.com/run-llama/llama_index/blob/1d861a9440cdc9...
So the system prompt is just part of the first prompt in a conversation? How is that different from not having a system prompt?
Earlier quoted context omitted.
would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg
Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.
The developer is also pretty responsive and actively looking for feedback (which is why it's currently free on TestFlight)
Does it save all conversations and let me revisit them later? I use MLC Chat to run Mistral 7B on my iPhone at the moment, but the lack of conversation history is a real nuisance: https://apps.apple.com/us/app/mlc-chat/id6448482937
Are these apps using the neural compute parts of Apple's chips? Or ar they just using the regular CPU/GPU cores?
TL;DR: No, nearly all these apps will use GPU (via Metal), or CPU, not Neural Engine (ANE). Why? I suggest a few main reasons: 1) No Neural Engine API 2) CoreML has challenges modeling LLMs efficiently right now. 3) Not Enough Benefit (For the Cost... Yet!) This is my best understanding based on my own work and research for a local LLM iOS app. Read on for more in-depth justifications of each point! --- 1) No Neural…
What would be the downside to padding all inputs to have consistent input token size?
Earlier quoted context omitted.
would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg
Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.
My understanding is Metal is currently causing hangs on devices when there is barely enough RAM to fit the model and prompt, but not quite enough to run. Will work on falling back to CPU to avoid this kind of experience much more aggressively than today.
Thank you for taking the time to both try it out and to share your experience; I will use it to ensure it’s better in the future.
Earlier quoted context omitted.
[flagged]
What the hell are you on about? Apple has rules and guidelines on (user) generated content, GP was asking whether it applied here.
Does it save all conversations and let me revisit them later? I use MLC Chat to run Mistral 7B on my iPhone at the moment, but the lack of conversation history is a real nuisance: https://apps.apple.com/us/app/mlc-chat/id6448482937
you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg
edit: I can't reply to you below: Do you have the right app, there's no TestFlight just App Store link - if it's ChatOnMac then it should have a dropdown at the top of the chat room to select a model. If it's empty or otherwise bugged out please let me know what you see in the top menu. It filters the available model presets based on how much RAM you have available, so let me know what specific device you have and I can look into it. Thank you.
The model presets are also configurable by forking the bot and loading your own via GitHub (bots run inside sandboxed hidden webviews inside the app). But this is not ergonomically friendly just yet.
Earlier quoted context omitted.
EDIT: Attempting to converse with any Q4_K_M 7B parameter model on a 15 Pro Max... the phone just melts down. It feels like it is producing about one token per minute. MLC-Chat can handle 7B parameter models just fine even on a 14 Pro Max, which has less RAM, so I think there is an issue here. EDIT 2: Even using StableLM, I am experiencing a total crash of the app fairly consistently if I chat in one conversation, th…
Wow, thanks so much for taking the time to test it out and share such great feedback! Thrilled about all those developments! More model options as well as link-based GGUF downloads on the way. On the 7b models: I’m very sorry for the poor experience. I wouldn’t recommend 7b over Q2_K at the moment, unless you’re on a 16GB iPad (or an Apple Silicon Mac!). This needs to be much clearer, as you observed the consequences…