Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PrivateGPT

github.com

Post reply on HN
Print view View original

31–40 of 147 posts

Re: PrivateGPT

#31

Post by lynx23 »

flagrant_taco wrote: post #18
dragonwriter wrote: post #7

Earlier quoted context omitted.

A less anthropomorphic approach might be to say that LLMs can predict the correct “shape” of an answer even when they don’t have data that gives them a clear right answer for the correct content, and since their basic design is to provide the best response they can, they’ll provide an answer of the correct shape with fairly random content if all they have good information to predict is the shape and not the content.

What's the point of the technology if it will provide an answer regardless of the accuracy? And what prevents this from being dangerous when the factual and ficticious answers are indistinguishable?

These are the 1-million dollar questions when it comes to LLMs. How useful is it to talk to a human who likes to talk, and prefers to say something over admiting they dont know? And if you have a person with münchhausensyndrome in your circles, how dangerous is it to listen to them and accidentally picking up a lie? LLMs with temp > 0.5 are effectively like these people.

Re: PrivateGPT

#33

Post by Grimblewald »

jstarfish wrote: post #4
simonw wrote: post #2

I'm always interested in seeing the prompt that drives these kinds of tools. In this case it appears to be using RetrievalQA from LangChain, which I think is this prompt here: https://github.com/hwchase17/langchain/blob/v0.0.176/langcha... Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. {context} Questio…

Do such fail-early conditions save processing time?

the incidence of "i don't know" in response to questions in the training data is pretty low if present at all, and even if it were you'd still need to frame those I don't know answers such that they apply to the entire dataset accurately. This is obviously a gargantuan undertaking that would not scale well as data is added, and so right now the idea or concept of not knowing something is not taught. At best you'd build a model that handles human language really well then retrieves information from a database and uses in context learning to answer questions, where a failure to find info results in an i don't know.

What is taught indirectly though is level of certainty, so if you get LLM's to rationalise their answers you tend to get more reliable evidence based answers.

Bottom line, teaching a monolithic model what it means to not know something with certainty, is difficult and not currently done. You'll likely get a lot of false negatives.

Re: PrivateGPT

#34

Post by rain1 »

awestroke wrote: post #32
skykooler wrote: post #10

"System requirements" section should really mention what amount of RAM or VRAM is needed for inference.

That depends on which model you use it with. It's "bring your own model"

so list a few known to work models and their requirements

Re: PrivateGPT

#35

Post by daitangio »

Hi, very interesting... what are the memory/disk requirements to run it? 16GB of RAM would be enough? I suggest to add these requirements to the README

Re: PrivateGPT

#36

Post by sinandrei91 »

Is there a benchmark for retrieval from multiple ft documents? I tried the LangchainQA with Pinecone and wasn't impressed with the search result when using it on my Zotero library.

Re: PrivateGPT

#37

Post by MandieD »

daitangio wrote: post #35

Hi, very interesting... what are the memory/disk requirements to run it? 16GB of RAM would be enough? I suggest to add these requirements to the README

Also, a general formula for estimating how much additional storage space will be claimed per MB/million words ingested would be helpful.

Re: PrivateGPT

#38

Post by santiagobasulto »

aldarisbm wrote: post #8

One quick plug I want to have the memory part of langchain down, vector store + local database + client to chat with an LLM (gpt4all model can be swapped with OpenAI api just switching the base URL) https://github.com/aldarisbm/memory It's still got ways to go, if someone wants to help let me know :)

Sorry for my ignorance. But memory refers to the process of using embeddings for QA right?

The process roughly is:

Ingestion:

- Process embeddings for your documents (from text to array of numbers)

- Store your documents in a Vector DB

Query time:

- Process embeddings for the query

- Find documents similar to the query using distance from other docs in the Vector db

- Construct prompt with format:

""" Answer question using this context: {DOCUMENTS RETRIEVED}

Question: {question} Answer: """

Is that correct? Now, my question is, can the models be swapped easily? Or that requires a complete recalculation of the embedding (and new ingestion)?

Re: PrivateGPT

#39

Post by dragonwriter »

flagrant_taco wrote: post #18
dragonwriter wrote: post #7

Earlier quoted context omitted.

A less anthropomorphic approach might be to say that LLMs can predict the correct “shape” of an answer even when they don’t have data that gives them a clear right answer for the correct content, and since their basic design is to provide the best response they can, they’ll provide an answer of the correct shape with fairly random content if all they have good information to predict is the shape and not the content.

What's the point of the technology if it will provide an answer regardless of the accuracy? And what prevents this from being dangerous when the factual and ficticious answers are indistinguishable?

> What's the point of the technology if it will provide an answer regardless of the accuracy?

The purpose is to serve as a component of a system which also includes features, such as the prompt structure upthread, that mitigates the undesired behavior while keeping the useful behaviors.

Post reply on HN