Got this working locally - badly needs GPU support (have a 3090 so come on!) there is some workaround but expect it will come pretty soon. This video was a useful walkthough esp on using different model and upping the CPU threads. https://www.youtube.com/watch?v=A3F5riM5BNE
PrivateGPT
71–80 of 147 posts
Re: PrivateGPT
#72Self-hosted + self-trained LLMs are probably the future for enterprise. While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META. They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products…
I suspect the major cloud providers will also each offer their own “enterprise friendly” LLM services (Azure already offers a version of OpenAI’s API). If they have the right data guarantees, that’ll probably be sufficient for companies that are already using their IaaS offerings.
Re: PrivateGPT
#73This will still hallucinate, right? Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?
How do you define hallucination?
factually incorrect / nonsensical output
Re: PrivateGPT
#74Earlier quoted context omitted.
Do such fail-early conditions save processing time?
If you mean the "If you don't know" part, oh no, they have a much bigger problem they're solving. The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that. LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie. Is this anthropomorphizing? Yep. But that's…
> Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them.
My take? It's like a high-schooler being asked a question by the teacher and having to answer on the spot. If they studied the material well, they'll give a good and correct answer. If they (like me, more often than I'd care to admit) only half-listened to the lectures and maaaaybe skimmed some cliff's notes before class, they will give an answer too - one strung together out of few remembered (or misremembered) facts, an overall feel for the problem space (e.g. writing style, historical period, how people behave), with lots and lots of interpolation in between. Delivered confidently, it has more chance of avoiding a bad mark (or even scoring a good one) than flat-out saying, "I don't know".
Add to that some usual mistakes out of carelessness and... whatever it is that makes you forget a minus sign and realize it half a page of equations later - and you get GPT-4. It's giving answers like a person who just blurts out whatever thoughts pop into their head, without making a conscious attempt at shaping or interrogating them.
Re: PrivateGPT
#75Earlier quoted context omitted.
A less anthropomorphic approach might be to say that LLMs can predict the correct “shape” of an answer even when they don’t have data that gives them a clear right answer for the correct content, and since their basic design is to provide the best response they can, they’ll provide an answer of the correct shape with fairly random content if all they have good information to predict is the shape and not the content.
What's the point of the technology if it will provide an answer regardless of the accuracy? And what prevents this from being dangerous when the factual and ficticious answers are indistinguishable?
We have the same problem with people. Somehow, we've managed to build a civilization that can, occasionally, fly people to the Moon and get them back.
Even if LLMs never get any more reliable than your average human, they're still valuable because they know much more than any single human ever could, run faster, only eat electricity, and can be scaled up without all kinds of nasty social and political problems. That's huge on its own.
Or, put another way, LLMs are kind of an concentrated digital extract of human cognitive capacity, without consciousness or personhood.
Re: PrivateGPT
#76Wow. I keep a personal Wiki, Journal and use plain text accounting... This project could help me create a personal AI which answers any questions to my life, finances or knowledge...
Well maybe it works on Obsidian vaults for note taking heh, but with llama models' 2k input token range it'd get a tenth of the way before starting to drop context. Likely useless without something like an 100k model.
Well you wouldn't input the whole Vault to the model, you would use embeddings to find the content that is most relevant to the question being asked.
Re: PrivateGPT
#77Earlier quoted context omitted.
If you mean the "If you don't know" part, oh no, they have a much bigger problem they're solving. The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that. LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie. Is this anthropomorphizing? Yep. But that's…
I think of it more like a pachinko machine. You put your question in the top, it bounces around through a bunch of biased obstacles, but intevitably it will come out somewhere at the bottom. By telling it not to lie to you, you're biasing it toward a particular output in the event that its confidence is low. Otherwise, low confidence results just fall out somewhere mostly random.
> By telling it not to lie to you, you're biasing it toward a particular output in the event that its confidence is low.
This is something I really don't understand about LLMs. I think I understand how the generative side of them work, but "asking" it to not lie baffles me. LLMs require a massive corpus of text to train the model, how much of that text contains tokens that translate to "don't lie to me", and scores well enough to make its way into the output?
Re: PrivateGPT
#78Re: PrivateGPT
#79Granted I'm not coming from the python world, but I have tried many of these projects, and very few of them install out of the box. They usually end with some incompatibility, and files scattered all over the place, leading to future nightmares. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.…
I would say it's more an artifact of historical tech debt that is hard to change now without breaking everyone. As another commenter pointed out, you want to use a venv - I use pipenv as a tool to automate this but there are others as well (poetry is probably better but pipenv seems to work for me).
Re: PrivateGPT
#80Granted I'm not coming from the python world, but I have tried many of these projects, and very few of them install out of the box. They usually end with some incompatibility, and files scattered all over the place, leading to future nightmares. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.…
You don’t need Docker, you just need a virtual env for each random thing you try instead of making them all conflict with each other. Maybe some day pip will add a switch to automatically create one, but until then, python3 -m venv venv . venv/bin/activate before you try something random. Also, `python` is usually Python 2.7. If it is, I advise removing it from your system unless you have a strong reason to keep it.
Sorry, but
(base) vid@kk:~/D/ai/tortoise-tts$ python3 -m venv venv
(base) vid@kk:~/D/ai/tortoise-tts$ . venv/bin/activate
(venv) (base) vid@kk:~/D/ai/tortoise-tts$ python -m pip install -r ./requirements.txt
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting rotary_embedding_torch
Using cached rotary_embedding_torch-0.2.3-py3-none-any.whl (4.5 kB)
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 196, in
setup_package()
File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 147, in setup_package
from numpy.distutils.core import setup
ModuleNotFoundError: No module named 'numpy'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
(venv) (base) vid@kk:~/D/ai/tortoise-tts$
This isn't the first time venv didn't work for me, then there's anaconda, miniconda, and a bunch of other things that add env and directories. I don't really know what any of them do, and -I don't want to- I'm not an expert on every app on my system, but I can use nearly all of them without pain. (remember this is about ease of use)
Oh yeah, and python 2 vs python 3.
It's very much the "works for me" experience from the old days. There's no good learning from it, except dependencies suck and python systems aren't good at them.
I think when releasing anything that includes dependencies that span the operating system, it's just good engineering to use a container approach. Otherwise you're just causing a lot of discomfort in the world for no good reason.
It's funny because chatgpt would give me an answer to this in a few moments, but I'm locked out for a week because it can't figure out where I am.
Now I'm spending my Sunday morning setting up a dockerfile for tortoise-tts. At least I will learn something reusable from that. I guess I will create a PR for it, though it seems the author isn't tending the repo anymore.