Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

apps.apple.com

Post reply on HN
Print view View original

171–180 of 208 posts

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#171

Post by Const-me »

coder543 wrote: post #167
Const-me wrote: post #166

Earlier quoted context omitted.

These data center targeted GPUs can only output that many tokens per second for large batches. These tokens are shared between hundreds or even thousands of users concurrently accessing the same server. That’s why despite these GPUs deliver very high throughput in tokens/second, responses do not appear instantly, and individual users observe non-trivial latency. Another interesting consequence, running these ML model…

> These data center targeted GPUs can only output that many tokens per second for large batches. No… my RTX 3090 can output 130 tokens per second with Mistral on batch size 1. A more powerful GPU (with faster memory) should easily be able to crack 200 tokens per second at batch size 1 with Mistral. At larger batch sizes, the token rate would be enormous. Microsoft’s high performing Phi-2 model breaks 200 tokens per s…

I wonder are you using a quantized version of Mistral? NVidia 3090 has 936 GB/second memory bandwidth, so 150 tokens/second = 7.2 GB per token. In the original 16 bits format, the model takes about 13GB.

Anyway, while these datacenter servers can deliver these speeds for a single session, they don’t do that because large batches result in much higher combined throughput.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#172

Post by freedomben »

alchemist1e9 wrote: post #158
freedomben wrote: post #143

Earlier quoted context omitted.

Apple forcefully takes on that responsibility, and their customers love it (as clearly evidenced by their domination of the market). If you don't want a hammer that gets reviewed and screened by Apple before you can use it, then you're on the wrong platform.

Yeah that’s fine. But we still don’t have any tangible specifics in regards to what Apple needs to be reviewing for a locally hosted LLM. What can possibly be the criteria? Probably an alternative version of this app or a similar app can provide an option to load your own models. Is that a problem for Apple to allow? What I have found in my personal (and perhaps biased and anecdotal experience) is that there is a lar…

Couldn't agree more.

I'm not sure how Apple could make an explicit policy for this. My theory is that they won't, but rather are going to roll out their own LLM that run locally and is optimized for on-device hardware, which non-Apple code will not be able to use. This won't make all the LLMs go away, but it will make running very unattractive since they'll be battery hungry and slow compared to the official app.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#173

Post by brittlewis12 »

smcleod wrote: post #134
brittlewis12 wrote: post #22

Earlier quoted context omitted.

you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg

This is really nice to use. Especially compared to MLC. Well done!

Thank you so much for taking the time to try it out!

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#174

Post by coder543 »

Const-me wrote: post #171
coder543 wrote: post #167

Earlier quoted context omitted.

> These data center targeted GPUs can only output that many tokens per second for large batches. No… my RTX 3090 can output 130 tokens per second with Mistral on batch size 1. A more powerful GPU (with faster memory) should easily be able to crack 200 tokens per second at batch size 1 with Mistral. At larger batch sizes, the token rate would be enormous. Microsoft’s high performing Phi-2 model breaks 200 tokens per s…

I wonder are you using a quantized version of Mistral? NVidia 3090 has 936 GB/second memory bandwidth, so 150 tokens/second = 7.2 GB per token. In the original 16 bits format, the model takes about 13GB. Anyway, while these datacenter servers can deliver these speeds for a single session, they don’t do that because large batches result in much higher combined throughput.

> I wonder are you using a quantized version of Mistral?

Yes, we’re comparing phone performance versus datacenter GPUs. That is the discussion point I was responding to originally. That person appeared to be asking when phones are going to be faster than datacenters at running these models. Phones are not running un-quantized 7B models. I was using the 4-bit quantized models, which are close to what phones would be able to run, and a very good balance of accuracy vs speed.

> Anyway, while these datacenter servers can deliver these speeds for a single session, they don’t do that because large batches result in much higher combined throughput.

I don’t agree… batching will increase latency slightly, but it shouldn’t affect throughput for a single session much if it is done correctly. I admit it probably will have some effect, of course. The point of batching is to make use of the unused compute resources, balancing compute vs memory bandwidth better. You should still be running through the layers as fast as memory bandwidth allows, not stalling on compute by making the batch size too large. Right?

We don’t see these speeds because datacenter GPUs are running much larger models, as I have said repeatedly. Even GPT-3.5 Turbo is huge by comparison, since it is believed to be 20B parameters. It would run at about a third of the speed of Mistral. But, GPT-4 is where things get really useful, and no one knows (publicly) just how huge that is. It is definitely a lot slower than GPT-3.5, which in turn is a lot slower than Mistral.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#175

Post by kccqzy »

antirez wrote: post #140

I see the efforts required to create the little app, but inference via llama.cpp or core Ml is trivial and the models are open weights, so it makes more sense to have a free app for this: most of the value is in the LLM which is free.

I'd argue in this space besides the model weights, a lot of the value comes from a nice, not-too-fancy but nevertheless intuitive and delightful UI. I mean I've used the free MLC Chat app which runs Mistral 7B fine, and because it's free, I have very low expectations of its UI design. If someone is making a new app with a nicer UI, I really don't mind paying a buck or two.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#176

Post by Const-me »

coder543 wrote: post #174
Const-me wrote: post #171

Earlier quoted context omitted.

I wonder are you using a quantized version of Mistral? NVidia 3090 has 936 GB/second memory bandwidth, so 150 tokens/second = 7.2 GB per token. In the original 16 bits format, the model takes about 13GB. Anyway, while these datacenter servers can deliver these speeds for a single session, they don’t do that because large batches result in much higher combined throughput.

> I wonder are you using a quantized version of Mistral? Yes, we’re comparing phone performance versus datacenter GPUs. That is the discussion point I was responding to originally. That person appeared to be asking when phones are going to be faster than datacenters at running these models. Phones are not running un-quantized 7B models. I was using the 4-bit quantized models, which are close to what phones would be a…

People use batching on servers to optimize throughput for the complete server, not for a single session.

See “throughput (tokens/s) versus concurrency” graph in that article: https://www.predera.com/blog/mistral-7b-performance-analysis...

There’re other interesting graphs there, they also measured the latency. They found a very strong dependency between batch size and latency, both for first token i.e. pre-fill, and time between subsequent tokens. Note how batch size = 40 delivers best throughput in tokens/second for the server, however the first output token takes almost 4 seconds to generate, probably too slow for an interactive chat.

BTW, I used development tools in the browser to measure latency for the free ChatGPT 3.5, and got about 900 milliseconds till the first token. OpenAI probably balanced throughput versus latency very carefully because their user base is large, and that balance directly affects their costs.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#177

Post by coder543 »

Const-me wrote: post #176
coder543 wrote: post #174

Earlier quoted context omitted.

> I wonder are you using a quantized version of Mistral? Yes, we’re comparing phone performance versus datacenter GPUs. That is the discussion point I was responding to originally. That person appeared to be asking when phones are going to be faster than datacenters at running these models. Phones are not running un-quantized 7B models. I was using the 4-bit quantized models, which are close to what phones would be a…

People use batching on servers to optimize throughput for the complete server, not for a single session. See “throughput (tokens/s) versus concurrency” graph in that article: https://www.predera.com/blog/mistral-7b-performance-analysis... There’re other interesting graphs there, they also measured the latency. They found a very strong dependency between batch size and latency, both for first token i.e. pre-fill, and…

The chart you pointed out is very interesting, but it largely supports my point.

The blue line is easiest to read, so let’s look at how the tokens/sec scale for a single user session as the batch size increases. It starts out at about 100 tokens/s for 5 users = 20 tokens/s/user. At the next point, it is about 19t/s/u. Beyond this point, we start losing some ground, but even by the final data point, it is still over 11t/s/u.

The throughput is affected by less than 2x even with the most unreasonably large batch size. (Unreasonable, because the time to first token is unacceptable for an interactive chat, as you pointed out.)

But, with a batch size that is balanced appropriately, the throughput for a single user session is effectively unchanged whether the service is batching at N=3 or N=10. (Or presumably N=1, but the chart doesn’t include that.) The time to first token is also a reasonable 1 second delay, which is similar to what OpenAI is providing in your testing.

So, with the right batching balance, batching increases the total throughput of the server, but does not affect the throughput or latency for any individual session very much. It does have some impact, of course. Model size and quantization seem to have a much larger impact than batching, from an end user standpoint.

Re: I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

#179

Post by ngai_aku »

wahnfrieden wrote: post #109
brittlewis12 wrote: post #22

Earlier quoted context omitted.

you can absolutely access and continue all your past chats in cnvrs! would love to hear what you think: https://testflight.apple.com/join/ERFxInZg

My free / mostly open source app also stores conversation history, synced via iCloud https://ChatOnMac.com edit: I can't reply to you below: Do you have the right app, there's no TestFlight just App Store link - if it's ChatOnMac then it should have a dropdown at the top of the chat room to select a model. If it's empty or otherwise bugged out please let me know what you see in the top menu. It filters the available…

I was excited when I saw this, but I'm having trouble with it (and it looks like I'm not the only one). As others have pointed out, the download link on your site does open TestFlight. I've since deleted that version and installed the official version from the AppStore after revisiting this thread in search of answers.

I now have the full version installed on my iPhone 15 pro, and I have added my OpenAI key, but none of the models I've selected (3.5 Turbo, 4, 4 Turbo) work. My messages in the chat have a red exclamation next to them which opens an error message stating 'Load failed' when clicked. If I click 'Retry Message' the entire app crashes.

Post reply on HN