While many NLP related Apple ML job listings have been added since this article was written, there were several recent listings at the time of its writing. While I feel that Apple does not focus well on intangible technologies, products that can't be readily carried, worn and given their boutique product development fetish focus, I have some hope that they can overcome this bias somewhat, and see how behind they are.
The Coming of Local LLMs
41–50 of 214 posts
Re: The Coming of Local LLMs
#42I know it's secret how GPT-4 actually works but are there any of these local LLMs that are also multimodal?
Re: The Coming of Local LLMs
#43While many NLP related Apple ML job listings have been added since this article was written, there were several recent listings at the time of its writing. While I feel that Apple does not focus well on intangible technologies, products that can't be readily carried, worn and given their boutique product development fetish focus, I have some hope that they can overcome this bias somewhat, and see how behind they are.
I disagree with the assessment that Apple is behind. Apple is known for executing well and putting their weight behind the things they launch.
Like Jack Dorsey would often say "it's not important to be first to market, you can just be best to market". And the world got CashApp.
I'm sure however Apple enters the space, it will be fleshed out (vs Bard).
Re: The Coming of Local LLMs
#44I expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable…
Low precision: agreed Attention: Isn't it quadratic in context length? I dunno, this feels like the crude first iteration of something that will get inevitably passed by something that scales better.
In theory one could use the nysromformer. Don't know if anyone does in practice
Re: The Coming of Local LLMs
#45I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…
LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.
Re: The Coming of Local LLMs
#46How realistic is CPU-only inference in the near future?
[deleted]
Re: The Coming of Local LLMs
#47I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…
Can you explain what you did set up wise for your test? I'm following this "space" but the exact, simple pipelines are eluding me.
Re: The Coming of Local LLMs
#48While many NLP related Apple ML job listings have been added since this article was written, there were several recent listings at the time of its writing. While I feel that Apple does not focus well on intangible technologies, products that can't be readily carried, worn and given their boutique product development fetish focus, I have some hope that they can overcome this bias somewhat, and see how behind they are.
I disagree with the assessment that Apple is behind. Apple is known for executing well and putting their weight behind the things they launch. Like Jack Dorsey would often say "it's not important to be first to market, you can just be best to market". And the world got CashApp. I'm sure however Apple enters the space, it will be fleshed out (vs Bard).
I want to agree, but it's pretty easy to find instances where Apple has dabbled but not delivered best-in-class solutions. Siri. iCloud. Home automation.
Re: The Coming of Local LLMs
#49As I see these things come out, it feels like there's not a lot of discussion on which hardware (that isn't one of the fancy new Macs?) As in, there might be a lot of graphics cards out there that could be used here? Is it only Nvidia still, is AMD a possibility? Maybe I'm missing something on how the tech works?
Re: The Coming of Local LLMs
#50Earlier quoted context omitted.
are we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?
Inference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine
Checkout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.