Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

201–210 of 214 posts

Re: The Coming of Local LLMs

#201

Post by dns_snek »

ingenieroariel wrote: post #63

In order to run large language models, we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.) and putting Linux on it to remove the artificial restrictions. Yes, we will be running them soon in low end hardware, but we need to get at least to GPT-3.5-turbo level of inference speed and quality before we try to make it small. I already started. $ neofetch -` x@dec…

Is this a joke or an ad? Paying Apple hardware premiums (for hardware that isn't well supported by PyTorch, et al.) just to load Linux onto it and run AI models in a closet somewhere, all seems like an incredibly wasteful way to go about it.

Re: The Coming of Local LLMs

#202

Post by muyuu »

SV_BubbleTime wrote: post #186
muyuu wrote: post #180

Earlier quoted context omitted.

you are having a good laugh if you think the average granny is going to do that anytime soon

Yea, lol, what was I thinking!? It’s not like my granny uses TLS ECDH SHA1 right!? Oh, wait, everyday she does? The tools adapt; email has never had to.

the tools will only adapt when a large number of people have got burnt

as always happens

Re: The Coming of Local LLMs

#203

Post by d4rkp4ttern »

whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

Agreed. I’ve said it elsewhere: how do we know that ClosedAI has fully published all their tricks used to build GPT3.* or later? The main “clues” people have are from the InstructGPT paper, but it should shock nobody if it turns out that paper reveals less than 10% of their techniques, which may have taken years to come up with. Competitors would need to rediscover those on their own or come up with new ideas. Simply repeating the ideas from that paper is likely not going to get them to a competitive model performance.

Re: The Coming of Local LLMs

#204

Post by ChatGTP »

heyoni wrote: post #192
ChatGTP wrote: post #174

Earlier quoted context omitted.

Some negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?

Your comment has been detected to contain oppositional tones and has thus been filtered out. You are hereby removed from the discourse. /s

Love it!

Re: The Coming of Local LLMs

#205

Post by anthlax »

whimsicalism wrote: post #98
dpflan wrote: post #16

Earlier quoted context omitted.

Sharing a comment from a similar line of discussion: """ Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once. Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Mayb…

There is literally no way that GPT-4 is using classic O(N^2) attention exclusively. They haven't released the results, but I promise you this is not what they are using exclusively.

Interesting! How do you know this?

Re: The Coming of Local LLMs

#206

Post by leetharris »

anentropic wrote: post #14

Apple should get working on a version of the Neural Engine that is useful for these models, and remove the 3GB size limit [1] to take full advantage of the 'unified' memory architecture. Game changer. Waste of die space currently (on Macbook at least, I'm sure they find uses for it in the iPhone) [1] https://github.com/smpanaro/more-ane-transformers/blob/main/...

It's not a waste on Mac, it will dynamically switch between GPU and NPU whenever CoreML is called. There are a decent amount of applications that use CoreML.

But I do agree it should be improved!

Re: The Coming of Local LLMs

#208

Post by pwinnski »

sliken wrote: post #178
pwinnski wrote: post #122

Earlier quoted context omitted.

Apple has two values in conflict with each other, I think. On the one hand, they want to deliver best-in-class solutions. On the other hand, they have a commitment to user privacy[0] as perhaps only a gay man growing up in the south might value. Siri should be better! It lost features post-acquisition by Apple, and it seems like user privacy is why. Home automation is arguable. If you consider a single point of failu…

Up until recently I'd have agreed on Apple's. Increasingly Apple seem to be blocking tracking, noteably from facebook, to make the most profit of that tracking. I've read claims that apple made between $5B and $20B on advertising in 2022. It's far from clear that Apple's view on privacy is going to stay the same.

That they're blocking tracking is still privacy-focused. There are unsubstantiated claims that they exempt themselves from the same tracking, but reports of their advertising revenue doesn't contribute anything to those claims.

At Apple's scale, it's relatively easy for them to deliver $20B in ad revenue without any privacy-invading means.

People seem to have forgotten, but ads used to be based on context, so people looking at apps related to fitness might see ads related to fitness, but that wouldn't follow them around when they looked at other things. Apple still seems to be doing that; I haven't seen fitness ads on games, or game ads on fitness apps.

Re: The Coming of Local LLMs

#209

Post by ChatGTP »

duckmysick wrote: post #199
ChatGTP wrote: post #174

Earlier quoted context omitted.

Some negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?

That's ok. If it's a good idea, somebody else will talk about in a positive way. Also, unless you are reading every comment in every thread, you are going to miss a few interesting ideas anyway. That's ok too.

Stay calm and keep summarising!

Re: The Coming of Local LLMs

#210

Post by skinner_ »

agumonkey wrote: post #111

Were transformers used in other contexts ? biochemistry ? geometry .. whatever.

Alphafold uses something they call Evoformer, it is an attention mechanism. Our group has tried, and so far failed to utilize transformers for a very very specific search problem in geometry (https://bit.ly/unit-distances).
Post reply on HN