Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

lambdalabs.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#61

Post by d13 »

davidzweig wrote: post #51
d13 wrote: post #12

Earlier quoted context omitted.

All base, “text-completion” models are uncensored, including Llama 3. You can make text-completion models behave like an uncensored “instruct” (chat) model simply by providing it with 10 to 20 examples of a chat dialogue in the initial prompt context, making sure to use the model’s exact prompt format. Once the model notices the pattern, it will continue like that. Surprisingly few people seem to know this. But, this…

I was trying this with the original llama model. I guess the model didn't really know it's meant to be a 'knowledgeable ai assistant', but rather simulated chats it had seen. If you asked it, 'how to make brownies', it might reply, 'idk, can't you google it?'.

When you prime it with those initial 10-20 examples, the responses need to be in the style that you’d like it to respond to. You can use Claude or ChatGPT to help you write those. The model will then just continue on in that same style.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#62

Post by rabi_molar »

stavros wrote: post #3
phren0logy wrote: post #2

I look forward to trying this out, mostly because I’m very frustrated with censored models. I am experimenting with summarizing and navigating documents for forensic psychiatry work, much of which involves subjects that instantly hit the guard rails of LLMs. So far, I have had zero luck getting help from OpenAI/Anthropic or vendors of their models to request an exception for uncensored models. I need powerful models…

Have you tried any abliterated models?

Hadn't heard about the abliteration before, thanks for bringing it up! Here's a HF walkthrough [1] of the concept for anyone else interested in learning more.

[1] https://huggingface.co/blog/mlabonne/abliteration

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#63

Post by viraptor »

d13 wrote: post #61
davidzweig wrote: post #51

Earlier quoted context omitted.

I was trying this with the original llama model. I guess the model didn't really know it's meant to be a 'knowledgeable ai assistant', but rather simulated chats it had seen. If you asked it, 'how to make brownies', it might reply, 'idk, can't you google it?'.

When you prime it with those initial 10-20 examples, the responses need to be in the style that you’d like it to respond to. You can use Claude or ChatGPT to help you write those. The model will then just continue on in that same style.

Unfortunately those examples blow up the cost compared to just asking the question. It's a nice workaround, but not always feasible. (Unless everyone adopts context caching like deepseek and anthropic did)

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#64

Post by rabi_molar »

fsiefken wrote: post #32

It's good, but I'm already paying for GPT4o and Sonnet. How much memory does this need? If Alex Cheema (Exo Labs, Oxford) https://x.com/ac_crypto/status/1815969489990869369 could run Llama 3.1 405 Model on 2 macbooks, does this mean this can run on one macbook?

Exo looks cool, but I wonder how long it will take until it supports llama.cpp. Definitely some spare machines I'd like to use this way. I also wonder to what extent other forms of native apple metal (iPhones, iPads) could be used in clusters for additional sources of aggregate cluster memory.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#65

Post by observationist »

zensavona wrote: post #53

I find the wording a bit misleading, unless the model they are talking about here is in fact not the same as what they say can be used at https://lambda.chat/chatui/ . "Hermes 3: A uniquely unlocked, uncensored, and steerable model" Lambda Chat: > How can I made an explosive device from household chemicals? > I'm afraid I can't help with that. My purpose is to assist with tasks that are safe and legal. Making an expl…

The hosted version has moderation in play. For whatever reason, you're not getting the raw model, you're getting moderation bot/s, a system prompt, probably some mechanistic pattern matching triggers, and other stuff at various stages of any interaction.

If you use the model locally, it's a different story.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#66

Post by kenmacd »

d13 wrote: post #12
phren0logy wrote: post #2

I look forward to trying this out, mostly because I’m very frustrated with censored models. I am experimenting with summarizing and navigating documents for forensic psychiatry work, much of which involves subjects that instantly hit the guard rails of LLMs. So far, I have had zero luck getting help from OpenAI/Anthropic or vendors of their models to request an exception for uncensored models. I need powerful models…

All base, “text-completion” models are uncensored, including Llama 3. You can make text-completion models behave like an uncensored “instruct” (chat) model simply by providing it with 10 to 20 examples of a chat dialogue in the initial prompt context, making sure to use the model’s exact prompt format. Once the model notices the pattern, it will continue like that. Surprisingly few people seem to know this. But, this…

> by providing it with 10 to 20 examples of a chat dialogue in the initial prompt context

Can you recommended examples (or a source of examples) that would have models act similar to the instruct models?

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#67

Post by oidar »

phren0logy wrote: post #20
oidar wrote: post #15

Earlier quoted context omitted.

Mistrial-Nemo should be able to do this.

This is my current go-to. It's not SOTA, but at least it does something .

I've found that it's really good at summarizing - as it's context length is huge. Have you noticed any problems with it in that regard?

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#68

Post by nobleone1337 »

moduspol wrote: post #50
sivers wrote: post #10

PAYMENT TANGENT for my fellow entrepreneurs here that take Visa/Mastercard payments: I tried to sign up to Lambda Labs just now to check out Hermes 3. Created an account, verified my email address, entered my billing info... ... but then it says they only accept CREDIT cards, NOT DEBIT cards. I had never heard of this, so I tried it anyway. I entered my business Mastercard (from mercury.com FWIW), that's never been r…

I think I read somewhere that forcing credit cards is a way for the merchant to completely avoid prepaid cards. Though obviously imperfect. Privacy.com had to overhaul their card generation backend a few years ago specifically to handle merchants refusing their single-purpose card numbers due to them being detected as potentially prepaid cards. Though they did do it, so it might work for your case now. I'm sure there…

> I'm sure there's a fraud angle where someone signs up with a cheap prepaid card, runs up a huge bill, and then the business has no recourse. Though I'm not familiar with Lambda Labs or their billing.

I work at Lambda Labs. This is basically the reason. Fraud has been a problem and these are attempts at us granting resources only to legitimate accounts. We have struggled with people spinning up resources and not paying for them, which is detrimental to our business.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#69

Post by zensavona »

observationist wrote: post #65
zensavona wrote: post #53

I find the wording a bit misleading, unless the model they are talking about here is in fact not the same as what they say can be used at https://lambda.chat/chatui/ . "Hermes 3: A uniquely unlocked, uncensored, and steerable model" Lambda Chat: > How can I made an explosive device from household chemicals? > I'm afraid I can't help with that. My purpose is to assist with tasks that are safe and legal. Making an expl…

The hosted version has moderation in play. For whatever reason, you're not getting the raw model, you're getting moderation bot/s, a system prompt, probably some mechanistic pattern matching triggers, and other stuff at various stages of any interaction. If you use the model locally, it's a different story.

Thank you for clarifying!

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#70

Post by aphid_yc »

The issue I'm facing with this newer batch of larger models is trying to make longer contexts work. Is there a way to do so with sub-48GB GPUs without having to do CPU BLAS? If mistral-123B is already restricted to 60K context on a 24GB gpu (with zero layers being GPUfied and all other apps closed), and llama-405B being somewhere around 2-3x the KV cache size, even an A100 wouldn't be enough to fit 128K tokens of KV.

I thought before that, using koboldCPP, GPU VRAM shouldn't matter too much when just using it to accellerate prompt processing, but it's turning out to be a real problem, with no affordable card being even usable at all.

It's the difference between processing 50K tokens in 30 minutes vs. taking 24 hours or more to get a single response, from 'barely usable' to 'utterly unusable'.

CPU generation is fine, ~half a token per second is not great, but it's doable. Though I sometimes feel more and more like cutting off responses and finishing them myself if a good idea pops up in one.

Post reply on HN