Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PrivateGPT

github.com

Post reply on HN
Print view View original

81–90 of 147 posts

Re: PrivateGPT

#81

Post by quaintdev »

cddotdotslash wrote: post #72
j_shi wrote: post #69

Self-hosted + self-trained LLMs are probably the future for enterprise. While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META. They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products…

I suspect the major cloud providers will also each offer their own “enterprise friendly” LLM services (Azure already offers a version of OpenAI’s API). If they have the right data guarantees, that’ll probably be sufficient for companies that are already using their IaaS offerings.

Enterprises should work on an open source LLM and run it on their own. This also helps people like you and me to run LLM at home.

It has worked before like in case of Linux and can work again.

Re: PrivateGPT

#82

Post by seydor »

joebiden2 wrote: post #66
seydor wrote: post #56

does this only work with llamaCPP ? I.e. can't use GPU models with this?

llama supports GPU by now: https://old.reddit.com/r/LocalLLaMA/comments/13gok03/llamacp...

How do i enable GPU in privateGPT w llamaCpp ? It turns my CPU to a vacuum cleaner

Re: PrivateGPT

#83

Post by XCSme »

divan wrote: post #73
XCSme wrote: post #60

Earlier quoted context omitted.

How do you define hallucination?

factually incorrect / nonsensical output

I assume this is only possible if the training data contains only a "right answer". If the training data contains two contradicting answers A and B, then, from the AIs perspective, there is no correct answer.

I assume that for questions like "What year was Bill Gates born in?", it should never return a wrong answer, if the answer was in the training data. If it was not, it should respond that it doesn't know.

Re: PrivateGPT

#84

Post by seydor »

davidy123 wrote: post #65

Granted I'm not coming from the python world, but I have tried many of these projects, and very few of them install out of the box. They usually end with some incompatibility, and files scattered all over the place, leading to future nightmares. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.…

This is too much to ask for an OSSLM project , considering that it will be obsolete by something else in 7 days or less.

Re: PrivateGPT

#85

Post by oefrha »

davidy123 wrote: post #80
oefrha wrote: post #67

Earlier quoted context omitted.

You don’t need Docker, you just need a virtual env for each random thing you try instead of making them all conflict with each other. Maybe some day pip will add a switch to automatically create one, but until then, python3 -m venv venv . venv/bin/activate before you try something random. Also, `python` is usually Python 2.7. If it is, I advise removing it from your system unless you have a strong reason to keep it.

Sorry, but (base) vid@kk:~/D/ai/tortoise-tts$ python3 -m venv venv (base) vid@kk:~/D/ai/tortoise-tts$ . venv/bin/activate (venv) (base) vid@kk:~/D/ai/tortoise-tts$ python -m pip install -r ./requirements.txt Collecting tqdm Using cached tqdm-4.65.0-py3-none-any.whl (77 kB) Collecting rotary_embedding_torch Using cached rotary_embedding_torch-0.2.3-py3-none-any.whl (4.5 kB) … × python setup.py egg_info did not run suc…

Oh I don't disagree, the ecosystem does has a packaging reproducibility and multitenancy problem with out-of-box tooling, and projects seldom provide basic instructions for people outside the ecosystem, like using a virtual env.

That said, this tortoise-tts project might be a particularly bad example. It somehow locks to scipy 0.10.1 from 2012 [1] (during the Python 3.2 release cycle, when Python 3 was heavily in flux) in requirements.txt [2]. Probably not terribly surprising it doesn't work. I didn't bother to look into why they lock to that.

[1] https://pypi.org/project/scipy/0.10.1/

[2] https://github.com/neonbjb/tortoise-tts/blob/0ea829d37aa6528...

Re: PrivateGPT

#86

Post by seydor »

simonw wrote: post #2

I'm always interested in seeing the prompt that drives these kinds of tools. In this case it appears to be using RetrievalQA from LangChain, which I think is this prompt here: https://github.com/hwchase17/langchain/blob/v0.0.176/langcha... Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. {context} Questio…

The problem is, when does it know that it does not know.

Re: PrivateGPT

#87

Post by gwd »

mabbo wrote: post #6
jstarfish wrote: post #4

Earlier quoted context omitted.

Do such fail-early conditions save processing time?

If you mean the "If you don't know" part, oh no, they have a much bigger problem they're solving. The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that. LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie. Is this anthropomorphizing? Yep. But that's…

> Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them.

I think it might be more accurate to say, "LLMs are writing a novel in which a very smart AI answers everyone's questions." If you were writing a sci fi novel with a brilliant AI, and you knew the answer to some question or other, you'd put in the right answer. But if you didn't know, you'd just make up something that sounded plausible.

Alternately, you can think of the problem as the AI taking an exam. If you get an exam question you're a bit fuzzy on, you don't just write "I don't know". You come up with the best answer you can given the scraps of information you do know. Maybe you'll guess right, and in any case you'll get some partial credit.

The first one ("writing a novel") is useful I think in contextualizing emotions expressed by LLMs. If you're writing a novel where some character expresses an emotion, you aren't experiencing that emotion. Nor is the LLM when they express emotions: they're just trying to complete the text -- i.e., write a good novel.

Re: PrivateGPT

#88

Post by monkeydust »

cddotdotslash wrote: post #72
j_shi wrote: post #69

Self-hosted + self-trained LLMs are probably the future for enterprise. While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META. They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products…

I suspect the major cloud providers will also each offer their own “enterprise friendly” LLM services (Azure already offers a version of OpenAI’s API). If they have the right data guarantees, that’ll probably be sufficient for companies that are already using their IaaS offerings.

How do the data rights broadly differ between OpenAI API directly and through Azure's endpoint?

Re: PrivateGPT

#89

Post by Art9681 »

divan wrote: post #59

This will still hallucinate, right? Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?

Like many others, I’m also building my own platform to accomplish this. What I’ve learned is the document preparation is key in getting the LLM to answer correctly. The text splitting portion is a crucial step here. Picking the correct splitter and parameters for your use case is important. At first I was getting incorrect or made up answers. Setting up a proper prompt template and text splitting parameters fixed the issue for the most part and now I have 99% success.

Also, the local model used makes a big difference. Right now wizard-mega and manticore are the best ones to use. I run the 16b ggml versions in an M2 Pro and it takes about 30 seconds to “warm up” and produce some quality responses.

Re: PrivateGPT

#90

Post by RcouF1uZ4gsC »

davidy123 wrote: post #65

Granted I'm not coming from the python world, but I have tried many of these projects, and very few of them install out of the box. They usually end with some incompatibility, and files scattered all over the place, leading to future nightmares. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.…

llama.cpp has revolutionized running these LLMs because it provides a nice, self-contained minimal dependency way to do this.

Python is very fragile to deploy and run on your own machine.

Post reply on HN