> For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. Did they go over the entire text with a thesaurus? I've never seen "palletization" be used as a viable synonym for "quantization" before, and I've read quite a few papers on LLM quantization
Apple's On-Device and Server Foundation Models
231–240 of 562 posts
Re: Apple's On-Device and Server Foundation Models
#232For people interested in AI research, there's nothing new here. IMO they should do a better job of referencing existing papers and techniques. The way they wrote about "adaptors" can make it seem like it's something novel, but it's actually just re-iterating vanilla LoRA. It was enough to convince one of the top-voted HackerNews comments that this was a "huge development". Benchmarks are nice though.
I think the thing they're saying that's novel, isn't what they have (LoRAs), but where and when and how they make them. Rather than just pre-baking static LoRAs to ship with the base model (e.g. one global "rewrite this in a friendly style" LoRA, etc), Apple seem to have chosen a bounded set of behaviors they want to implement as LoRAs — one for each "mode" they want their base model to operate in — and then set up a…
Easel has been on iMessage for a bit now: https://apps.apple.com/us/app/easel-ai/id6448734086
Re: Apple's On-Device and Server Foundation Models
#233Earlier quoted context omitted.
RAM is literally the cheapest primary component in a laptop at going rate of 1-4usd/GB. I'd say that shipping 8GB base model in 2024 is clearly manipulation by Apple, i.e. planned obsolescence or a way to moat Apple software. Anyone who doesn't see this is just being delusional. Same way Apple and Samsung ship 128GB of storage when the production price between 128gb and 1tb is like 10$ (on a 1000$ device). Samsung ev…
> RAM is literally the cheapest primary component Is that still true for Apple's integrated memory? It might be - I just don't know.
For the LPDDR4 and LPDDR5 that goes into the M1 and M2/M3 systems, yes. You might need to spend more money on memory controllers (since M1 and up is 8-channel) but the physical memory component itself is highly availible and relatively cheap. Same goes for SSD storage, nowadays.
Re: Apple's On-Device and Server Foundation Models
#234Easel on iMessage has had this experience plus more for a while, including multiplayer, where you can have two people in one scene together with photorealistic imagery: https://apps.apple.com/us/app/easel-ai/id6448734086
Re: Apple's On-Device and Server Foundation Models
#235Earlier quoted context omitted.
This gives me the vibe of calling high resolution screens as "retina" screens.
I don't see anything wrong with that at all. They've created a branding term that allows consumers to get an idea of the sort of pixel density they can expect without having to actually check, should they not want to bother.
Agreed. It is not high resolution as such, but high resolution that the user can relate to - like cannot see the pixel.
Still remember the hard time using Apple newton in a conference vs the palm freely on loan in a Gartner group conference. Palm solved a problem, even though not very Apple … user can input on a small device. I kept it, on top of my newly bought newton.
It is the user …
Re: Apple's On-Device and Server Foundation Models
#236It’s interesting that a sub-ChatGPT 3.5 class model can do a lot of things on-device if you marry it with a good platform and feed it personal context. GPT-4o, living on the browser, is not as compelling as a product compared to what Apple Intelligence can do on the iPhone with a less capable model.
Re: Apple's On-Device and Server Foundation Models
#237For people interested in AI research, there's nothing new here. IMO they should do a better job of referencing existing papers and techniques. The way they wrote about "adaptors" can make it seem like it's something novel, but it's actually just re-iterating vanilla LoRA. It was enough to convince one of the top-voted HackerNews comments that this was a "huge development". Benchmarks are nice though.
I thought the news of them using Apple Silicon rather than NVIDIA in their data centers was significant.
Perhaps there is still hope of a relaunch of xserve; with the widespread use of Apple computers amongst developers Apple has a real chance of challenging NVIDIA's CUDA moat.
Re: Apple's On-Device and Server Foundation Models
#238Earlier quoted context omitted.
Without the "so we can spy on you" part.
But they won't even make good on that : https://arstechnica.com/tech-policy/2023/12/apple-admits-to-... There's your bleeding, sorry truth there. It's only a matter of time until we get another headline like it.
There's a difference between being forced to compromise user security and doing it willingly in the name of vague "AI safety" concerns.
Furthermore, most governments don't like the "march in with a warrant and demand information" approach, because it's loud and noisy. People might move data out of a given cloud if they know there's spooks inside. And more importantly, it creates a paper trail, which they don't want. So there's a lot of effort put into compromising cloud servers by intelligence agencies.
Looking at Apple's blog post regarding Private Cloud Compute[0], they've basically took every security precaution they could to prevent covert compromise of their servers. They also have some fancy attestation stuff that, most notably, creates a paper trail whenever software changes. Once again, spooks absolutely hate this. It's technically possible for Apple to subvert this scheme, but that would require coordination from several different business units at Apple. Which, again, creates a paper trail. Spooks would much rather exploit a vulnerability than demand code signing keys that would provide evidence of cooperation.
To be clear: no, this isn't end-to-end. You can't currently do end-to-end encrypted cloud compute[1]. But it's still Apple putting lots of money into a significant improvement in terms of privacy and transparency regarding cloud services. OpenAI in contrast does not give two flying fucks about your data privacy, and makes building an AI Panopticon one of their deliberate, expressly stated design goals. Their safety team, at least by their own admission, cannot operate without total knowledge of everything their models get prompted with so they can implement reactive controls for specific exploits.
[0] https://security.apple.com/blog/private-cloud-compute/
[1] Homomorphic encryption is not theoretically impossible, but imposes significant performance penalties that negate the performance advantages of Apple using a cloud service. I suspect that they at least gave it some thought though.
Re: Apple's On-Device and Server Foundation Models
#239Earlier quoted context omitted.
For everything other than handwriting I don't think the LoRAs are fine-tuned locally.
Well, here's another one: they promised that your local (non-iCloud) photos don't leave the device. Yet they will now — among many other things they mentioned doing with your photos — allow you to generate "Memoji" that look like the people in your photos. Which includes the non-iCloud photos. I can't picture any way to use a RAG to do that. I can picture a way to do that that doesn't involve any model fine-tuning, b…
I believe this could be achieved by providing a seed image to the diffusion model and generating memoji based on it. This way fine tuning isn't required.
Re: Apple's On-Device and Server Foundation Models
#240it would have been nice if they allowed you to build your own apple AI system (i refused to redefine apples AI as just AI :-p ) using clusters of mac minis and mac pros.
but of course they still want that data for themselves like google does. its secure against everyone but apple and the NSA probably lol.