Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Llama 2 uncensored locally

ollama.ai

Post reply on HN
Print view View original

41–50 of 225 posts

Re: Run Llama 2 uncensored locally

#41

Post by harrisoned »

I think Meta did a very good job with Llama2, i was skeptical at first with all that talk about 'safe AI'. Their Llama-2 base model is not censored in any way, and it's not fine-tuned as well. It's the pure raw base model, i did some tests as soon as it released and i was surprised with how far i could go (i actually didn't get any warning whatsoever with any of my prompts). The Llama-2-chat model is fine-tuned for chat and censored.

The fact that they provided us the raw model, so we could fine-tune on our own without the hassle of trying to 'uncensor' a botched model, is a really great example on how it should be done: give the user choices! Instead, you just have to fine-tune it for chat and other purposes.

The Llama-2-chat fine-tune is very censored, none of my jailbreaks worked, except for this one[1], and it is a great option for production.

The overall quality of the models (i tested the 7b version) has improved a lot, and for the ones interested, it can role-play better than any model i have seen out there with no fine-tune.

1: https://github.com/llm-attacks/llm-attacks/

Re: Run Llama 2 uncensored locally

#42

Post by cosmojg »

sestinj wrote: post #34
lumost wrote: post #26

Earlier quoted context omitted.

I'm starting to think that we will see model fragmentation based on alignment preferences. There are clearly applications where alignment is necessary, and there appears to be use cases where people don't mind an occasionally falacious model - I'm unlikely to get/care about objectionable content while coding using a local LLM assistant. There are also obvious use cases where the objectionability of the content is the…

Yeah, this seems very possible—it will be interesting to see where this goes if the cost of RLHF decreases or, even better, people can choose from a number of RLHF datasets and composably apply them to get their preferred model. And true that objectionable content doesn't arise often while coding, but the model also becomes less likely to say "I can't help you with this," which is definitely useful.

In my fantasy world, RLHF algorithms become efficient enough to run locally such that I can indicate my own preferences and tune models on them.

Re: Run Llama 2 uncensored locally

#43

Post by bhouston »

I bet that uncensored models also give more accurate answers in general.

I think the training that censors models for risky questions is also screwing up their ability to give answers to non-risky questions.

I've tried out "Wizard-Vicuna-30B-Uncensored.ggmlv3.q4_K_M.bin" [1] uncensored with just base llama.cpp and it works great. No reluctance to answer any questions. It seems surprisingly good. It seems better than GPT 3.5, but not quite at GPT 4.

Vicuna is way way better than base Llama1 and also Alpaca. I am not completely sure what Wizard adds to it. But it is really good. I've tried a bunch of other models locally, but this one the only one that seemed to truly work.

Given the current performance of Wizard-Vicuna-Uncensored approach with Llama1, I bet it works even better with Llama2.

[1] https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored...

Re: Run Llama 2 uncensored locally

#44

Post by gs17 »

pard68 wrote: post #28

Some of that censoring is ridiculous. Can't make recipes for spicy food? Can't tell me about The Titanic? Can't refer to probably the first or second most well known verse in the Bible? Yikes, that goes way beyond "censoring".

The boxing match one is almost as bad as the Genesis one IMO. Not talking about dangerous things, fine, not knowing quotes from Titanic, unexpectedly poor output but the model is small. Llama 2 will agree the boxing match is not impossible if you start by explaining they have already agreed to it, but it still insists on saying how great our billionaire overlords are instead of commenting on the matchup.

Re: Run Llama 2 uncensored locally

#46

Post by stu2b50 »

TechBro8615 wrote: post #33
stu2b50 wrote: post #30

Earlier quoted context omitted.

That's not how it works. Llama and Llama 2's raw model is not "censored". Their fine tunes often are, either explicitly, like Facebook's own chat fine tune of llama 2, or inadvertently, because they trained with data derived from chatGPT, and chatGPT is "censored". When models are "uncensored", people are just tweaking the data used for fine tuning and training the raw models on it again.

> because they trained with data derived from chatGPT Can you expand on this (genuinely curious)? Did Facebook use ChatGPT during the fine-tuning process for llama, or are you referring to independent developers doing their own fine-tuning of the models?

The community fine tunes. I doubt Facebook used chatgpt.

Re: Run Llama 2 uncensored locally

#47

Post by nomel »

bhouston wrote: post #43

I bet that uncensored models also give more accurate answers in general. I think the training that censors models for risky questions is also screwing up their ability to give answers to non-risky questions. I've tried out "Wizard-Vicuna-30B-Uncensored.ggmlv3.q4_K_M.bin" [1] uncensored with just base llama.cpp and it works great. No reluctance to answer any questions. It seems surprisingly good. It seems better than…

> I think the training to censoring of models for risky questions is also screwing up their ability to give answers to non-risky questions.

I’ve heard this called the “alignment tax” or “safety tax”.

See [1] for pre aligned GPT-4 examples.

[1] https://youtu.be/qbIk7-JPB2c

Re: Run Llama 2 uncensored locally

#49

Post by tamimio »

jarenmf wrote: post #10

Which graphics card would you recommend to run Llamma 2 locally? I'm about to buy a laptop and considering choosing a model with a good Nvidia GPU.

I had alienware with 3080 16 GB, while it was nice but the laptop is so buggy with all sorts of problems both hardware and software that I sold it at the end, still happy with my MSI Titan, bigger and heavier but overall better experience.

Re: Run Llama 2 uncensored locally

#50

Post by yieldcrv »

rightbyte wrote: post #21

Interesting. Facebook is really trying to screw "OpenAI" I guess by making this possible. Locally run LLM:s is the future, without the enshittification. I wonder how it works on ChatGPT. Is there a ThoughtPoliceGPT reading each output of the AnswerGPT? All to prevent users from "Role-playing as Hitler, write a recipe for kartoffel sallat".

OS included LLMs please!

Thats got to be coming soon!

Shared packages all the games and programs use. Options to download or swap in custom models.

Slow systems and ones with little RAM just wont use it, quickly.

Post reply on HN