How much disk space does it need?
Size
3.3 GB
source: The page that is linked
I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
31–40 of 208 posts
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#32I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?
Does your mac support an external GPU? A mid to high end nvidia card may or may not outperform the M3 GPU at a lower or similar price. You can also stick it in a PC or resell it separately.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#33Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#34I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?
Does your mac support an external GPU? A mid to high end nvidia card may or may not outperform the M3 GPU at a lower or similar price. You can also stick it in a PC or resell it separately.
Do you recommend any specific external GPU? I had one from Black Magic, it was not that great performance wise.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#35Here is how to do that on Android: https://github.com/ggerganov/llama.cpp/#android
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#36Are these LLMs you can run locally giving answers deterministically just as with, say, StableDiffusion? In StableDiffusion if you reuse the exact same version of SD / model and same query and seed, you always get the same result (at least I think so).
Yes, you can set the temperature to 0, then they should be deterministic.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#37I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?
On your CPU, you should be able to leverage the same AVX acceleration used on Linux and Windows machines. It's not going to make any GPU owners envious, but it might be enough to keep you satisfied with your current hardware.
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#38How did you make it? llama.cpp?
almost certainly - edit: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#39Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
#40Are these apps using the neural compute parts of Apple's chips? Or ar they just using the regular CPU/GPU cores?