Will these smaller on device models lead to a crash in GPU prices?
Prices fall when supply outpaces demand -- this is adding more demand.
This isn't adding GPU demand.
Apple's On-Device and Server Foundation Models
51–60 of 562 posts
Re: Apple's On-Device and Server Foundation Models
#52Earlier quoted context omitted.
Does that imply I just stole your comment by reading it? No snark intended; I’m seriously asking. If the answer is “no” then where do you draw the line?
I don’t actually think this is complicated and reading a comment is not the same thing as scraping the internet and you obviously know that. A few factors that come to mind would be: - scale - informed consent which there was none in this case - how you are going to use that data. For example using everybody others work so the worlds richest company can make more money from it while giving back nothing in return is a…
I think it's even simpler than that: incentives. The entire premise of copyright law (and all IP law) is to protect the incentive to create new stuff, which is often a very risky and highly time or capital intensive endeavor.
So here's the question:
Does a person reading a comment destroy the incentive for the author to post it? No. In fact, it is the only thing that produces the incentive for someone to post. People post here when they want that thing to be read by someone else.
Does a model sucking up all the artistic output of the last 400 years and using that to produce an image generator model destroy the incentive of producing and sharing said artistic output? Yes. At least, that is the goal of such a model -- to become so good it is competitive with human artists.
Of course you have plenty of people positioned benefit from this incentive-destruction claiming it does no such thing. I personally tend to put more credence in the words of people who have historically actually been incentivized by said incentives (i.e. artists) who generally seem to perceive this as destructive to their desire to create and share their work.
Re: Apple's On-Device and Server Foundation Models
#53Earlier quoted context omitted.
From your own link: > Controlling data usage > In addition to following all robots.txt rules and directives, Apple has a secondary user agent, Applebot-Extended, that gives web publishers additional controls over how their website content can be used by Apple. > With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s foundation models powering generative AI fe…
But it also says that Applebot-Extended doesn't crawl webpages and instead this marker is only used to determine what can be done with the pages that were visited by Applebot. Not that I like an opt-out system, but based on the wording of the docs it is true that if you blocked Applebot then blocking Applebot-Extended isn't necessary.
Yeah that is true, but I suspect that most publishers that want their content to appear in search but not used for model training will not have blocked Applebot to date (hence the original commenter's argument)
Re: Apple's On-Device and Server Foundation Models
#54Earlier quoted context omitted.
No, they said they did. Huge difference
It was mentioned in the keynote that they allow researchers to audit their claims.
And as soon as independent sources support that they've made good on this claim it will be more than a claim. I actually am impressed by the link I missed and was provided elsewhere in this thread, and I hope to also be impressed when this claim is actually realized and we have more details about it
Re: Apple's On-Device and Server Foundation Models
#55Earlier quoted context omitted.
Might want to actually read it: Applebot-Extended does not crawl webpages. They gave this as an additional control to allow crawling for search but blocking for use in models.
> There is no AppleBot-Extended. And if you blocked it in the past it remains blocked. You said there is no Applebot-Extended. The link says otherwise.
It's still true that there's no Applebot-Extended if it isn't crawling pages. Rather it's a marker to ask Applebot to limit what it does with your pages.
Re: Apple's On-Device and Server Foundation Models
#56Earlier quoted context omitted.
Likely they’ll be able to take advantage of the hardware neural engine and be far more power efficient. Apple has demonstrated this is something it takes pretty seriously.
So iOS LLM Apps dont use the neural engine? Lol
If they use Llama.cpp they probably run on the GPU. Apple hasn’t published much about their neural engine, so you kinda have to use it through CoreML. I assume they have some aces up their sleeves for running LLMs efficiently that haven’t told anyone yet.
Re: Apple's On-Device and Server Foundation Models
#57“We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks.”
This is huuuuge. I don’t see announcement of 3rd party training support yet, but I imagine/hope it’s planned.
One of the hard things about local+private ML is I don’t want every app I download to need GBs of weights, and don’t want a delay when I open a new app and all the memory swap happens. As an app developer I want the best model that runs on each HW model, not one lowest common denominator model for slowest HW I support. Apple has the chance to make this smooth: great models tuned to each chip, adapters for each use case, new use cases only have a few MB of weights (for a set of current base models), and base models can get better over time (new HW and improved models). Basically app thinning for models.
Even if the base models aren’t SOTA to start, the developer experience is great and they can iterate.
Server side is so much easier, but look forward to local+private taking over for a lot of use cases.
Re: Apple's On-Device and Server Foundation Models
#58It would be interesting to see how these models impact battery life. I’ve tried a few local LLMs on my iPhone 15 Pro via the PrivateLLM app, and the battery charge plummets just after a few minutes of usage.
During my time at Apple the bigger issue with personalized, on-device models was the file size. At the time, each model was a significant amount of data to push to a device, and with lots of teams wanting an on-device model and the desire to update them regularly, it was definitely a big discussion.
They’ve gone with a single 3B model and several “adapters” for each use case. One adapter is good at summarising while another good a generating message replies.
Re: Apple's On-Device and Server Foundation Models
#59Earlier quoted context omitted.
Web scraping is legal. And if you run a website and want to opt-out then simply add a robots.txt. The standard way of preventing bots for 30 years.
How are people supposed to block it when they stole all the data first and then only after that point they decide to even tell anyone what user agent they need to block and how they are planning to exploit your work for their profit.
You just have a rule that says block everything except crawlers: A, B, C.
Also the AppleBot was known about before it appeared in Siri.
Re: Apple's On-Device and Server Foundation Models
#60> Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks. For our models we adapt the attention matrices, the attention projection matrix, and the fully connected layers in…
The A in LoRA stands for adapters
LoRA stands for "Low Rank Adaptation" btw.