Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Having a 20GB file that lets you ask an offline computer any question is amazing

old.reddit.com

Post reply on HN
Print view View original

11–20 of 127 posts

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#11

Post by samwillis »

I am completely convinced the future is local LLMs, privet, on your device. But then also fine tuned with your own data. All your documents, email, movements, browser history - maybe it even gets trained on your calls. Effectively an extension to your own brain.

I really don't want my AI assistant in the cloud, controlled by a corporation, reading and seeing everything I do. I would go as far as saying I want local AI more and open source AI.

Another comment said they want it to be an offline Wikipedia, we don't need that, we need it to know all our own stuff, that's what we work with every day.

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#13

Post by tetris11 »

behnamoh wrote: post #7

My goal: download the entire Wikipedia and store it in a vector db. Then have a local LLM answer questions based on Wikipedia. Best of both worlds!

I wonder what personality it would take on, when you ask it write fiction

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#14

Post by NicoJuicy »

noobermin wrote: post #4

The attitude in the comments is amazing. The LLM chatbot hype cycle bears a lot of resemblance to the crypto hype before the crash.

The utility of LLM's is a very different question as the profitablity of LLM's.

That was something that was severely overlapping with crypto. Most people wanted to believe it's useful because it would mean $$$

LLM 's are here to stay, that's for sure.

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#15

Post by kgeist »

noobermin wrote: post #4

The attitude in the comments is amazing. The LLM chatbot hype cycle bears a lot of resemblance to the crypto hype before the crash.

It seems like everyone in our company uses generative AI at the moment: marketing department coming up with ideas, designers creating illustrations, programmers writing code etc. I.e. there's a lot of useful real life application of generative AI right now, people actually use it for work. Crypto, for most part, in my opinion, did not provide as much value to the average person. And it's now possible to run LLMs locally, so you shouldn't be afraid of exit scams :)

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#16

Post by petesergeant »

noobermin wrote: post #4

The attitude in the comments is amazing. The LLM chatbot hype cycle bears a lot of resemblance to the crypto hype before the crash.

I would pay several hundred dollars a month to keep access to GPT-4, and never found any use at all for crypto. There might be hype, but there’s also an insane amount of utility

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#17

Post by moonchrome »

kramerger wrote: post #8

We can do that, but then we will need to close all applications to run it because it will need 20GB RAM or VRAM to run. It is still more practical to visit stackoverflow when I have a question. (still impressive how much this improves every week. Maybe in 2-3 months we have something more useful)

I got 64gb ram in my desktop for few hundred €. Laptops overprice RAM (as a price discriminator I guess).

But unlike Apple SoC, most GPUs have separate memory - getting a GPU with 24gb ram is very very expensive in comparison.

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#18

Post by the_third_wave »

behnamoh wrote: post #7

My goal: download the entire Wikipedia and store it in a vector db. Then have a local LLM answer questions based on Wikipedia. Best of both worlds!

Given the biased nature of many if not most Wikipedia articles it would be certain that such a system would bend the truth to an unacceptable degree. Train the thing on more than just Wikipedia, e.g add Encyclopedia Britannica to the training set. This is the one area where "diversity" really matters: diversity of opinion. Using a single biased source as your oracle will turn you into a pawn for those who control the bias.

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#19

Post by lastangryman »

noobermin wrote: post #4

The attitude in the comments is amazing. The LLM chatbot hype cycle bears a lot of resemblance to the crypto hype before the crash.

Within a few minutes you can sign up to OpenAI and see the potential of ChatGPT immediately, in you own hands. The potential of crypto/web3 was always an abstract idea (jumped on by so many scammers), there was nothing you could actually use to see the utility.

Re: Having a 20GB file that lets you ask an offline computer any question is amazing

#20

Post by narrator »

The thing about transformer models that are impressive is that they compress all sorts of stuff about human understanding down into a incredibly tiny space. They asked GPT-4 to explain GPT-2 and it found things like "something done right" or "metaphors" corresponded to neurons. It's this meta understanding of thought, just like diffusion models are this meta understanding of art. The real innovation is one can turn a crank and it keeps getting deeper and more subtle insights into the whole thing. If you turn that crank with A100s for a few days it gets very deep and doesn't hit a wall. In the earlier days of machine learning that didn't used to happen. Understanding was bounded at some point and didn't increase, but it seems transformers just keep going.
Post reply on HN