121–130 of 246 posts
The app looks great! Likewise, if you have any requests or ideas for improving llama.cpp, please don't hesitate to open an issue / discussion in the repo
Earlier quoted context omitted.
Can you speak more to this? I get useful output from LLMs all the time, but never use system prompts. What am I missing?
Sure, I use one system prompt template to make ChatGPT be more concise. Compare these two: https://sharegpt.com/c/fEZKMIy vs https://sharegpt.com/c/S2lyYON I use similar ones to get ChatGPT to be more thorough or diligent as well. From my limited experience with local models, this type of system prompting is even more important than with ChatGPT 4.
This is awesome. I currently use Ollama with OpenWebUI but am a big fan of native apps so this is right up my alley.
There is a demo clip with a vertical scroll bar which does not fade out as it would do in a native mac app:)
> Thanks to the amazing work of @ggerganov on llama.cpp which made this possible. If there is anything that you wish to exist in an ideal local AI app, I'd love to hear about it. The app looks great! Likewise, if you have any requests or ideas for improving llama.cpp, please don't hesitate to open an issue / discussion in the repo
Поздрави!
Earlier quoted context omitted.
Since I couldn't find it in your list, I'd like to plug my own macOS (and iOS) app: Private LLM. Unlike almost every other app in the space, it isn't based on llama.cpp (we use mlc-llm) or naive RTN quantized models (we use OmniQuant). Also, the app has deep integrations with macOS and iOS (Shortcuts, Siri, macOS Services, etc). Incidentally, it currently runs Mixtral 8x7B Instruct[2] and Mistral[3] models faster tha…
What's the performance like in tokens/s?
> Thanks to the amazing work of @ggerganov on llama.cpp which made this possible. If there is anything that you wish to exist in an ideal local AI app, I'd love to hear about it. The app looks great! Likewise, if you have any requests or ideas for improving llama.cpp, please don't hesitate to open an issue / discussion in the repo
Earlier quoted context omitted.
Yes it would be the next big focus on this. Personal data connectivity is what I see where local AI would excel - despite model power differences.
I have doubts about that. Most personal data actually lives in the cloud these days. If you need your Gmail emails, you'll need to use their API which is guarded behind $50k certification fee or so. I think there is a simpler version for personal use, but you still need to get the API key. Who's going to teach their mom about API keys? So I think for a lot of these data sources you'll end up with enterprise AIs integ…
Earlier quoted context omitted.
I have doubts about that. Most personal data actually lives in the cloud these days. If you need your Gmail emails, you'll need to use their API which is guarded behind $50k certification fee or so. I think there is a simpler version for personal use, but you still need to get the API key. Who's going to teach their mom about API keys? So I think for a lot of these data sources you'll end up with enterprise AIs integ…
I think this is a good take. While there's big enough niche for personal data locally, I'd love if there's a way to solve for email/cloud data requiring API keys.
Sadly I can't try this because I'm on Windows or Linux. Was testing apps like this if anyone is interested: Best / Easy to use: - https://lmstudio.ai - https://msty.app - https://jan.ai More complex / Unpolished UI: - https://gpt4all.io - https://pinokio.computer - https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generat... - https://github.com/LostRuins/koboldcpp Misc: - https://faraday.dev (AI Characters): No U…