Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Llama 2 uncensored locally

ollama.ai

Post reply on HN
Print view View original

201–210 of 225 posts

Re: Run Llama 2 uncensored locally

#201

Post by unshavedyak »

unshavedyak wrote: post #184
moffkalast wrote: post #158

Earlier quoted context omitted.

Yes it's become rather obvious when the fine tunes produced by the Wizard team perform worse on all benchmarks than Hartford's versions that are trained on the same dataset but with the refusals removed.

What specific Hartford versions are you referencing? A previous post was talking about how impressed they were with Wizard, and you’re saying Hartford is even better? You’ve got me curious! Hopefully it’s available in ggml

Nevermind, seems that is Hartford's version. Guess i'm just not savvy on the relationship between all the involved parties hah.

Re: Run Llama 2 uncensored locally

#202

Post by thangngoc89 »

brucethemoose2 wrote: post #16
jarenmf wrote: post #10

Which graphics card would you recommend to run Llamma 2 locally? I'm about to buy a laptop and considering choosing a model with a good Nvidia GPU.

A 16GB 3080 is probably the cheapest and most ideal in a big laptop. But you can get some acceleration with anything ~6GB and up.

Laptop RTX has half the VRAM comparing to their PC counterparts. So 3080 laptop has 8GB

Re: Run Llama 2 uncensored locally

#203

Post by kromem »

thamer wrote: post #91

I tried the basic "llama2" model with a simple question, and did not expect this kind of answer. Q: who was the first man on the moon? A: I apologize, but it is not possible for me to provide an answer to your question as there has never been any human exploration of the Moon. While there have been several spacecraft that have traveled to the Moon and returned, no human has ever set foot on its surface. The last time…

Just a tip - I forget where I saw it, but at some point in reading over research I saw that using 'Q' and 'A' results in lower accuracy than 'Question' and 'Answer.'

Which probably fits the latter biasing more towards academic sample test kind of situations as opposed to the former.

Re: Run Llama 2 uncensored locally

#204

Post by baobabKoodaa »

thewataccount wrote: post #81
harrisoned wrote: post #41

I think Meta did a very good job with Llama2, i was skeptical at first with all that talk about 'safe AI'. Their Llama-2 base model is not censored in any way, and it's not fine-tuned as well. It's the pure raw base model, i did some tests as soon as it released and i was surprised with how far i could go (i actually didn't get any warning whatsoever with any of my prompts). The Llama-2-chat model is fine-tuned for c…

I like the combination of releasing the raw uncensored + censored variants. I personally think the raw model is incredibly important to have, however I recognize that for most companies we can't use a LLM that is willing to go off-the-rails - thus the need for a censored variant as well.

Even those companies will not need the censored variant released by Meta. They will be better off running their own fine tunes.

Re: Run Llama 2 uncensored locally

#205

Post by benterix »

all2 wrote: post #164
wongarsu wrote: post #153

Earlier quoted context omitted.

I think God makes sense as a character, but only if you see him as a "person" with desires, flaws and some character development. If you treat him like some omnipotent, omniscient, unchanging, immutably good being (as some religious people like to do) you get into lots of contradictions

I'd be curious to hear about some of these contradictions. I've seen giant lists floating around, but after checking into some of them they seem spurious at best. I'm curious to know if you have something concrete?

You don't need any giant lists - earlier someone mentioned the love for widows and children, and yet this didn't seem to apply to random peoples who at a given time were the enemies of Israelites and were ordered to be murdered, including all women and children, no exceptions.

Re: Run Llama 2 uncensored locally

#206

Post by kromem »

Havoc wrote: post #117

I strongly dislike the current black/white stance around it being either censored or not. If someone wants to build a sexting bot...go for it & have fun. But stuff like engineering humanity ended viruses...yeah maybe suppressing that isn't the worst of ideas. Which puts us on a slippery slope of where to draw the line yes, but such is reality - a murky grey scale.

It's also a false sense of security.

A bad actor can sidestep alignment safety measures fairly successfully for the foreseeable future using dynamic jailbreaking efforts.

Good actors get penalized by the product being made notably worse to prevent that.

Perhaps a better approach would be having an uncensored model behind a discriminator trained to detect 'unsafe' responses and return an error if detected.

This would both catch jailbreaking by bad actors which successfully returned dangerous responses and accidentally dangerous responses.

But it would be far less likely to prevent a user asking for a dangerously spicy recipe from getting it.

There's an increased API cost because you are paying for two passes instead of one, but personally I'd rather pay 2x the cost for an excellent AI than half cost for a mediocre one.

Re: Run Llama 2 uncensored locally

#207

Post by BoorishBears »

huggingmouth wrote: post #55
nomel wrote: post #47

Earlier quoted context omitted.

> I think the training to censoring of models for risky questions is also screwing up their ability to give answers to non-risky questions. I’ve heard this called the “alignment tax” or “safety tax”. See [1] for pre aligned GPT-4 examples. [1] https://youtu.be/qbIk7-JPB2c

It's not suprising when you think what llms really are: when you "censor" them, you're forcing them to give output that doesn't "honestly" follow, essentially training them to give wrong information.

That's not how that works. Take some uncensored or "unaligned" models hallucinating racist things based on a name:

The default name for a person is John Doe. Anglo Saxon names in general are extremely common across the internet for non-nefarious reasons. So the tokens that make up "John" have a ton of associations in a wide variety of contexts and if the model hallucinates there's no particularly negative direction you'd expect it to go.

But Mohammed doesn't show up as often in the internet, and while it's also for non-nefarious reasons, it results in there being significantly fewer associations in the training data. What would be background noise for in the training data for John ends up being massively distorted by the smaller sample size: even tendencies for people to make racist jokes about the name.

-

People have this weird idea that OpenAI and co are aligning these models according to some hidden agenda but the reality is minorities are a minority of the training data for very obvious reasons. So if you don't "censor" them, you're not making them more truthful, you're leaving them dumber for a lot of tasks.

There's censorship beyond that which feels very CYA happening, but I really hope people aren't clamoring to sticking models that aren't intelligent enough to realize the tokens for John vs Mohammed should not affect a summarization task into anything even tangentially important...

Re: Run Llama 2 uncensored locally

#209

Post by bhouston »

BoorishBears wrote: post #207
huggingmouth wrote: post #55

Earlier quoted context omitted.

It's not suprising when you think what llms really are: when you "censor" them, you're forcing them to give output that doesn't "honestly" follow, essentially training them to give wrong information.

That's not how that works. Take some uncensored or "unaligned" models hallucinating racist things based on a name: The default name for a person is John Doe. Anglo Saxon names in general are extremely common across the internet for non-nefarious reasons. So the tokens that make up "John" have a ton of associations in a wide variety of contexts and if the model hallucinates there's no particularly negative direction y…

> But Mohammed doesn't show up as often in the internet, and while it's also for non-nefarious reasons, it results in there being significantly fewer associations in the training data. What would be background noise for in the training data for John ends up being massively distorted by the smaller sample size: even tendencies for people to make racist jokes about the name.

I do a lot of astrophotography - https://www.astrobin.com/users/bhouston/ Very often you do not have enough data of specific features you were trying to capture -- they are just too faint and close to the noise floor. The solution isn't for me to just go in and manually draw in photoshop in what I think it should look like though - that is just making up data - the solution is to get more data or leave it as it was captured.

I think it is the same thing with these LLM models. Do not make up data to fill in the gaps, show me what is really out there. And I will be a big boy about it and deal with it head on.

Re: Run Llama 2 uncensored locally

#210

Post by cubefox »

bhouston wrote: post #112
cubefox wrote: post #100

Earlier quoted context omitted.

> I bet that uncensored models also give more accurate answers in general. Doubtful: https://news.ycombinator.com/item?id=36976236 RLHF can motivate models to deny truths which are politically taboo, but it can also motivate them to care more about things supported by scientific evidence rather than about bullshitting, random conspiracy theories, and "hallucination". So it's a double edged sword.

I understand that it is the same technique for both. This makes sense. But to train a model to deny truths which are politically taboo does seem to be misaligned with training a model to favor truths, no? And what is taboo can be very broad if you want to make everyone happy. I would rather know the noble lie [1] is a lie, and then repeat it willing instead of not knowing it is a lie. My behavior in many situations w…

> But to train a model to deny truths which are politically taboo does seem to be misaligned with training a model to favor truths, no?

Strictly speaking, RLHF trains models to give answers which the human raters believe to be correct. In uncontroversial territory this correlates with truth, in taboo territory only with what is politically correct.

Post reply on HN