121–130 of 208 posts
Some other local LLM iOS apps: - MLC Chat: https://llm.mlc.ai - LLM Farm: https://llmfarm.site - Enchanted (not local, just a frontend): https://github.com/AugustDev/enchanted But I don't think any of these support Mistral 0.2 which is a pretty big deal.
Are these apps using the neural compute parts of Apple's chips? Or ar they just using the regular CPU/GPU cores?
TL;DR: No, nearly all these apps will use GPU (via Metal), or CPU, not Neural Engine (ANE). Why? I suggest a few main reasons: 1) No Neural Engine API 2) CoreML has challenges modeling LLMs efficiently right now. 3) Not Enough Benefit (For the Cost... Yet!) This is my best understanding based on my own work and research for a local LLM iOS app. Read on for more in-depth justifications of each point! --- 1) No Neural…
to prod you to sell yourself a bit more - what is the goal/selling point of cnvrs?
I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?
Earlier quoted context omitted.
I don't think running raw llama.cpp under termux in a shell on your phone, after downloading and compiling it from scratch,, is really comparable to 'I made an app'.
[flagged]
Earlier quoted context omitted.
A machine that tells you that the Golden State Warriors won the 2012 Stanley Cup by bowling a perfect 300 would be invaluable in 1602
In that case no need for a LLM, just a wikipedia dump with a full-text index is enough.
Earlier quoted context omitted.
No, system prompts are surrounded by [INST] [/INST]
No, every user input is surrounded by those tags. Scroll down from that link you posted and read the next two example prompts.
Here's the example from https://www.promptingguide.ai/models/mistral-7b#chat-templat...
[INST] What is your favorite condiment? [/INST] "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!" [INST] The right amount of what? [/INST]
Note that the sentence starting "Well, I'm quite partial isn't inside the tag.
This example comes from the official system card created by MistralAI available here: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
You can try it:
ollama run Mistral "[INST] What is your favorite condiment? [/INST] Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen [INST] The right amount of what? [/INST]"To think that we went from ClosedAi (OpenAi)s chatGPT to now being able to do this on out phones offline is incredible.