You can run it on Android too. Not very fast though.
Link?
Llama3 running locally on iPhone 15 Pro
51–59 of 59 posts
Re: Llama3 running locally on iPhone 15 Pro
#52Re: Llama3 running locally on iPhone 15 Pro
#53Earlier quoted context omitted.
I think it's silly to think the marketing department gets to control the pricing, but it is definitely very true that the "starting at " is very powerful for them. Even beyond Apple, it warps and distorts the entire laptop field pricing because people who don't understand how inadequate the entry level model is will compare that price to an entry-level model of Lenovo, or Dell, etc and make conclusions. Even on HN I'…
I don't think that they are inadequate, these devices are perfect for most of my family. They do some calls, messages, a couple of pictures here and there, basic word processing and web browsing but not much more on these devices. I had a Macbook with 8GB RAM and 256GB disk as my daily driver for work until last year running Docker and my fat IDE without too many issues. It's a similar story with my phone - I bought…
Interesting. what is the browser usage like for most of your family? i.e. how many tabs do they tend to keep open at a time?
Re: Llama3 running locally on iPhone 15 Pro
#54Earlier quoted context omitted.
Nice. What is battery life like under heavy use? I was reading a thread on the llama.cpp repo earlier where they were discussing whether it was possible (or attractive) to add neural engine support in some form.
With bigger 7B and 8B models, the battery life goes from a over a day to a few hours on my iPhone 15 Pro. The 8B model nominally works on 6GB phones but it's quite slow on them. OTOH, it's very usable on iPhone 15 Pro/ Pro Max devices and even better on M1/M2 iPads. Every framework: llama.cpp, MLX, mlc-llm (which I use) all only use the GPU. Using the ANE and perhaps the undocumented AMX coprocessor for efficient dec…
Super interesting, thank you!
Re: Llama3 running locally on iPhone 15 Pro
#55I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.
So the trend with Apple has been that the SoC from the current generation Pro and Pro Max devices becomes the SoC for the next generation of baseline devices. For instance the iPhone 14 Pro Max and iPhone 15 have the same SoC (A16 Bionic). And this trend holds all the way back to iPhone 12. It's almost certain that the iPhone 16 will ship with 8GB of RAM. What needs to be seen is whether iPhone 16 Pro and Pro Maxes w…
So I plotted iPhones RAM and I fail to see a trend that could lead to doubling current RAM to 16GB.
12 Pro Max had 6GB RAM
15 Pro Max has 8GB RAM
16 most likely will have between 8GB and 12GB RAM.
Re: Llama3 running locally on iPhone 15 Pro
#56new apple pro ai model with 32gb ram !!
Re: Llama3 running locally on iPhone 15 Pro
#57I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.
I can't wait until Groq or someone else release tiny mobile inference engines specifically for phones and the like.
There's already tiny LLMs for this. They're bad. Because it's not enough information to be coherent.
Re: Llama3 running locally on iPhone 15 Pro
#58This is quite impressive to be honest. The chat is answering at a speed of one word per few/several seconds. But still, this a nice feat. Example recording for the curious: https://www.youtube.com/watch?v=nZEvUj-QTrI
On my S24 Ultra, I am seeing it generate several words a second.
Re: Llama3 running locally on iPhone 15 Pro
#59Earlier quoted context omitted.
...so, you haven't used Android in over a decade?
Last time I cared how much RAM any phone had, iOS or Android, I was working at Augmentra on the ViewRanger app, and we were still supporting older devices with only 256 MB. That was… *checks CV*… I left in April 2015. I think RAM is like roads: usage expands to fill available infrastructure/storage. That an iPhone today has as much RAM as the still-functioning Mid-2013 MacBook Air sitting in a drawer behind me is sur…
I was basically always slowed down by RAM on Android - prob bc I switch between lots of very badly coded apps... so even on desktop I've grown to see RAM as "insurance against badly written code" as in "I'll still be able to run that memory leaky crapware and get what I need done" or in "I'll just spin up a VM for that crap that only runs on that other OS"...
Swimming in badly written SPAs and cordova/whatever hybrid apps is seriously helped by eg 12GB of RAM on a mobile :)