Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PrivateGPT

github.com

Post reply on HN
Print view View original

51–60 of 147 posts

Re: PrivateGPT

#51

Post by moffkalast »

daitangio wrote: post #35

Hi, very interesting... what are the memory/disk requirements to run it? 16GB of RAM would be enough? I suggest to add these requirements to the README

Well I'm not sure which models specifically work, but it runs on llama.cpp, which would mean lama derivative ones. Here's a little table for quantized CPU (GGML) versions and the RAM they require as a general rule of thumb:

> Name Quant method Bits Size RAM required Use case

WizardLM-7B.GGML.q4_0.bin q4_0 4bit 4.2GB 6GB 4bit.

WizardLM-7B.GGML.q4_1.bin q4_0 4bit 4.63GB 6GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.

WizardLM-7B.GGML.q5_0.bin q5_0 5bit 4.63GB 7GB 5-bit. Higher accuracy, higher resource usage and slower inference.

WizardLM-7B.GGML.q5_1.bin q5_1 5bit 5.0GB 7GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.

WizardLM-7B.GGML.q8_0.bin q8_0 8bit 8GB 10GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.

> Name Quant method Bits Size RAM required Use case

wizard-vicuna-13B.ggmlv3.q4_0.bin q4_0 4bit 8.14GB 10.5GB 4-bit.

wizard-vicuna-13B.ggmlv3.q4_1.bin q4_1 4bit 8.95GB 11.0GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.

wizard-vicuna-13B.ggmlv3.q5_0.bin q5_0 5bit 8.95GB 11.0GB 5-bit. Higher accuracy, higher resource usage and slower inference.

wizard-vicuna-13B.ggmlv3.q5_1.bin q5_1 5bit 9.76GB 12.25GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.

wizard-vicuna-13B.ggmlv3.q8_0.bin q5_1 5bit 16GB 18GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.

> Name Quant method Bits Size RAM required Use case

VicUnlocked-30B-LoRA.ggmlv3.q4_0.bin q4_0 4bit 20.3GB 23GB 4-bit.

VicUnlocked-30B-LoRA.ggmlv3.q4_1.bin q4_1 5bit 24.4GB 27GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.

VicUnlocked-30B-LoRA.ggmlv3.q5_0.bin q5_0 5bit 22.4GB 25GB 5-bit. Higher accuracy, higher resource usage and slower inference.

VicUnlocked-30B-LoRA.ggmlv3.q5_1.bin q5_1 5bit 24.4GB 27GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.

VicUnlocked-30B-LoRA.ggmlv3.q8_0.bin q8_0 8bit 36.6GB 39GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.

Copied of some of The-Bloke's model descriptions on huggingface. With 16G you can run practically all 7B and 13B versions. With shared GPU+CPU inference, one can also offload some layers onto a GPU (not sure if that makes the initial RAM requirement smaller), but you do need CUDA of course.

Re: PrivateGPT

#52

Post by nullsense »

lynx23 wrote: post #31
flagrant_taco wrote: post #18

Earlier quoted context omitted.

What's the point of the technology if it will provide an answer regardless of the accuracy? And what prevents this from being dangerous when the factual and ficticious answers are indistinguishable?

These are the 1-million dollar questions when it comes to LLMs. How useful is it to talk to a human who likes to talk, and prefers to say something over admiting they dont know? And if you have a person with münchhausensyndrome in your circles, how dangerous is it to listen to them and accidentally picking up a lie? LLMs with temp > 0.5 are effectively like these people.

The number one problem for generalized intelligence is establishing trust.

Re: PrivateGPT

#54

Post by bigfudge »

santiagobasulto wrote: post #38
aldarisbm wrote: post #8

One quick plug I want to have the memory part of langchain down, vector store + local database + client to chat with an LLM (gpt4all model can be swapped with OpenAI api just switching the base URL) https://github.com/aldarisbm/memory It's still got ways to go, if someone wants to help let me know :)

Sorry for my ignorance. But memory refers to the process of using embeddings for QA right? The process roughly is: Ingestion: - Process embeddings for your documents (from text to array of numbers) - Store your documents in a Vector DB Query time: - Process embeddings for the query - Find documents similar to the query using distance from other docs in the Vector db - Construct prompt with format: """ Answer question…

The embeddings can be based on a different model to the one you pass them as context to. So you could upgrade the summmariser model without upgrading the embeddings.

Re: PrivateGPT

#57

Post by andruby »

udev4096 wrote: post #45
dutchbrit wrote: post #44

Earlier quoted context omitted.

Day and time when you post something matters a lot unfortunately.

And now I am getting downvoted for it. Perfect

possibly because the tone of your post and that it doesn't actually add to the conversation.

weekday and time of day have an impact. Thousands of entries are posted each day (see https://news.ycombinator.com/newest) most never get a comment or upvote.

Re: PrivateGPT

#58

Post by santiagobasulto »

bigfudge wrote: post #54
santiagobasulto wrote: post #38

Earlier quoted context omitted.

Sorry for my ignorance. But memory refers to the process of using embeddings for QA right? The process roughly is: Ingestion: - Process embeddings for your documents (from text to array of numbers) - Store your documents in a Vector DB Query time: - Process embeddings for the query - Find documents similar to the query using distance from other docs in the Vector db - Construct prompt with format: """ Answer question…

The embeddings can be based on a different model to the one you pass them as context to. So you could upgrade the summmariser model without upgrading the embeddings.

But you'd need to keep both models in parallel, right? Using M1 to keep computing embeddings and using M2 for completions.

Re: PrivateGPT

#59

Post by divan »

This will still hallucinate, right?

Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?

Re: PrivateGPT

#60

Post by XCSme »

divan wrote: post #59

This will still hallucinate, right? Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?

How do you define hallucination?
Post reply on HN