I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros • Hacker News Board

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#131

Post by jwr » Mon, Jan 08, 2024, 10:17 AM UTC

geuis wrote: post #27

I have a 2020 16in MacBook Pro. I think it's the last generation of Intel chips. I've been struggling to get some of the LLM models like Mixtral to run on it. I hate the idea of needing to buy another $3k laptop less than 4 years after spending that much on my current machine. But if I want to get serious about developing non-chatgpt services, do I need a new M2 or M3 chip to get this stuff running locally?

Data point: my MacBook Pro 16" with the M3 Max (64GB) runs 34b model inference about as fast (or slightly faster) as ChatGPT runs GPT-4.

I am now running phind-codellama:34b-v2-q8_0 through ollama and the experience is very good.

All that said, though, every model I tried couldn't hold a candle to GPT-4: they all produce crappy results, aren't good at translation, and can't really do much for me. They are toys, I go "ooh" and "aah" over them, then realize they aren't that useful and go back to using GPT-4.

Perhaps 34B is still not enough to get anything resonable.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#132

Post by janfoeh » Mon, Jan 08, 2024, 10:38 AM UTC

wahnfrieden wrote: post #109

brittlewis12 wrote: post #22

Earlier quoted context omitted.

you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg

My free / mostly open source app also stores conversation history, synced via iCloud https://ChatOnMac.com edit: I can't reply to you below: Do you have the right app, there's no TestFlight just App Store link - if it's ChatOnMac then it should have a dropdown at the top of the chat room to select a model. If it's empty or otherwise bugged out please let me know what you see in the top menu. It filters the available…

> Do you have the right app, there's no TestFlight just App Store link

On chatonmac.com, the "Download on the App Store" button does not link the App Store for me either - I get a modal titled "Public Beta & Launch Day News" with "Join the TestFlight Beta" and "Launch Day Newsletter Signup Form".

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#133

Post by yreg » Mon, Jan 08, 2024, 10:51 AM UTC

pizza wrote: post #67

Zetobal wrote: post #59

Earlier quoted context omitted.

The base models of stablediffusion were always deterministic if you use a deterministic noise scheduler...

I think they’re referring to CUDA (and possibly other similar runtimes) being able to schedule floating point ops non-deterministically, combined with floating point arithmetic being potentially non-associative. I’m not personally sure how big an issue that would be for the output though.

I have never spotted any difference when regenerating (a recent) image with the same settings/seed/noise and I do it often. Haven't compared the bits though.

Older images are often difficult to reproduce for me - I believe due to changes in tooling (mostly updating Auto1111).

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#134

Post by smcleod » Mon, Jan 08, 2024, 11:33 AM UTC

brittlewis12 wrote: post #22

simonw wrote: post #2

Does it save all conversations and let me revisit them later? I use MLC Chat to run Mistral 7B on my iPhone at the moment, but the lack of conversation history is a real nuisance: https://apps.apple.com/us/app/mlc-chat/id6448482937

you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg

This is really nice to use. Especially compared to MLC. Well done!

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#135

Post by Const-me » Mon, Jan 08, 2024, 11:38 AM UTC

I made a free and open source Windows equivalent: https://github.com/Const-me/Cgml/releases/tag/1.1

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#136

Post by rrr_oh_man » Mon, Jan 08, 2024, 11:40 AM UTC

halJordan wrote: post #83

sp332 wrote: post #78

Earlier quoted context omitted.

It's $1.99 and the description says: The app requires a Pro iPhone with a minimum of 6GB of RAM. Only the following devices meet the requirement: - iPhone 15 Pro, iPhone 14 Pro, iPhone 13 Pro, iPhone 12 Pro. - iPads: Please check. RAM varies based on model and year.

So if they know it wont work, and do not put that info into the store's compatibility matrix then it's still a bait/switch to me. Compare to the Resident Evil page which does set the store limits on what devices can dl it.

It's clearly spelled out, App Store refunds work more often than they don't...

...and it's a $1.99 risk ffs.

Tangential point: It's super easy to go off the rails and on a rant, while the real reason behind someone's "bait/switch" is external, trivial, and benign. We tend to judge others by their actions, but ourselves by our intentions. I used a German company's excellent sleep supplement (and later worked for that company, too) which was being bashed on Facebook as "non FDA approved snake oil". Meanwhile, the FDA refused (and still refuses) to even look at anything outside actual drugs, even if you wanted them to. Sometimes your hands are just tied.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#137

Post by YetAnotherNick » Mon, Jan 08, 2024, 11:57 AM UTC

nl wrote: post #21

YetAnotherNick wrote: post #9

Earlier quoted context omitted.

Mistral instruct doesn't have system prompt AFAIK. Also llama chat system prompt is very useless in my testing.

Mistral Instruct does use a system prompt. You can see the raw format here: https://www.promptingguide.ai/models/mistral-7b#chat-templat... and you can see how LllamaIndex uses it here (as an example): https://github.com/run-llama/llama_index/blob/1d861a9440cdc9...

Look into their official page: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

There is no system prompt. Unless Llamaindex or some other sources cite something from mistral, I am inclined to believe they just copied it from llama.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#138

Post by Kelteseth » Mon, Jan 08, 2024, 12:01 PM UTC

Const-me wrote: post #135

I made a free and open source Windows equivalent: https://github.com/Const-me/Cgml/releases/tag/1.1

Can you link a ready to use example model I can just download and toy around with?

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#139

Post by Const-me » Mon, Jan 08, 2024, 12:19 PM UTC

Kelteseth wrote: post #138

Const-me wrote: post #135

I made a free and open source Windows equivalent: https://github.com/Const-me/Cgml/releases/tag/1.1

Can you link a ready to use example model I can just download and toy around with?

The model is on BitTorrent, see readme for the frontend app:

https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral...

The torrent file is also inside the MistralChat.zip archive.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#140

Post by antirez » Mon, Jan 08, 2024, 12:52 PM UTC

I see the efforts required to create the little app, but inference via llama.cpp or core Ml is trivial and the models are open weights, so it makes more sense to have a free app for this: most of the value is in the LLM which is free.