Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

lambdalabs.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#41

Post by itake »

simonw wrote: post #37
torginus wrote: post #23

Earlier quoted context omitted.

How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot and particular details, as well as asking how many times does a particular thing happen in the book, and the results have been hit and miss. I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.

Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.

I had issues with gpt4 last summer confusing which characters did what and hallucinating characters.

his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#42

Post by TradingPlaces »

fsiefken wrote: post #32

It's good, but I'm already paying for GPT4o and Sonnet. How much memory does this need? If Alex Cheema (Exo Labs, Oxford) https://x.com/ac_crypto/status/1815969489990869369 could run Llama 3.1 405 Model on 2 macbooks, does this mean this can run on one macbook?

I did not have great luck with 405b on an M1 Max Mac Studio with 64 GB of RAM via olama. I think memory was this issue. 70b runs with acceptable speed

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#43

Post by sweca »

itake wrote: post #41
simonw wrote: post #37

Earlier quoted context omitted.

Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.

I had issues with gpt4 last summer confusing which characters did what and hallucinating characters. his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.

Were you using the API or ChatGPT? ChatGPT uses RAG on large documents which is useless for many tasks.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#44

Post by smusamashah »

phren0logy wrote: post #2

I look forward to trying this out, mostly because I’m very frustrated with censored models. I am experimenting with summarizing and navigating documents for forensic psychiatry work, much of which involves subjects that instantly hit the guard rails of LLMs. So far, I have had zero luck getting help from OpenAI/Anthropic or vendors of their models to request an exception for uncensored models. I need powerful models…

I remember the Salesforce (edit: snowflake arctic instruct actually) LLM being completely uncensored and unaligned. Someone from the team said in response here on HN that they didn't do any alignment whatsoever.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#45

Post by miki123211 »

torginus wrote: post #23
phren0logy wrote: post #2

I look forward to trying this out, mostly because I’m very frustrated with censored models. I am experimenting with summarizing and navigating documents for forensic psychiatry work, much of which involves subjects that instantly hit the guard rails of LLMs. So far, I have had zero luck getting help from OpenAI/Anthropic or vendors of their models to request an exception for uncensored models. I need powerful models…

How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot and particular details, as well as asking how many times does a particular thing happen in the book, and the results have been hit and miss. I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.

For whatever it's worth, asking a model to count is a terrible idea due to how they work.

You may have more luck with a hybrid approach, using LLMs for language understanding and computers for the counting. For example, ask them to write a short, one-line description of every instance where something happens, and then use a traditional program to count the lines.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#46

Post by Lerc »

TradingPlaces wrote: post #42
fsiefken wrote: post #32

It's good, but I'm already paying for GPT4o and Sonnet. How much memory does this need? If Alex Cheema (Exo Labs, Oxford) https://x.com/ac_crypto/status/1815969489990869369 could run Llama 3.1 405 Model on 2 macbooks, does this mean this can run on one macbook?

I did not have great luck with 405b on an M1 Max Mac Studio with 64 GB of RAM via olama. I think memory was this issue. 70b runs with acceptable speed

64GB of ram is 550b bits of ram. That doesn't give you a lot of options on how many bits you want to keep per model parameter.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#47

Post by phren0logy »

torginus wrote: post #23
phren0logy wrote: post #2

I look forward to trying this out, mostly because I’m very frustrated with censored models. I am experimenting with summarizing and navigating documents for forensic psychiatry work, much of which involves subjects that instantly hit the guard rails of LLMs. So far, I have had zero luck getting help from OpenAI/Anthropic or vendors of their models to request an exception for uncensored models. I need powerful models…

How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot and particular details, as well as asking how many times does a particular thing happen in the book, and the results have been hit and miss. I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.

If you chunk the document well (which is surprisingly tricky with messy documents), then you can do summary via map reduce or other techniques.

To be clear: I don't trust it to provide an accurate summary; it's not meant to replace reading the documents. But it may help to find relevant parts of a document later, or get a reasonable overview of documents before starting a manual review. I expect (but will have to see) if it's better than just opening up a random PDF and starting there.

I recently had a case with over 24k pages of records, some of which were in PDFs that were thousands of pages long. The ability to do semantic search rather than keyword search was useful for when I said to myself, "Didn't I see something about that before? Where was that?"

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#48

Post by sroussey »

phren0logy wrote: post #47
torginus wrote: post #23

Earlier quoted context omitted.

How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot and particular details, as well as asking how many times does a particular thing happen in the book, and the results have been hit and miss. I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.

If you chunk the document well (which is surprisingly tricky with messy documents), then you can do summary via map reduce or other techniques. To be clear: I don't trust it to provide an accurate summary; it's not meant to replace reading the documents. But it may help to find relevant parts of a document later, or get a reasonable overview of documents before starting a manual review. I expect (but will have to see…

I’ve found summarizing this way is a great idea but the summarizing models are not great themselves. Would even prefer an embedding with the summary as maybe the embedding would be better.

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#49

Post by famouswaffles »

KTibow wrote: post #40
poisson-fish wrote: post #7

Earlier quoted context omitted.

try google's gemini models, safety filtering can be completely disabled via cloud studio or api

This is the 2nd time I've seen someone confused about this. The safety filtering is a layer on top of the model's built in ability to refuse, similar to OpenAI's "moderation" endpoint.

This is true but still Gemini is by far the least restrictive Sota LLM out there (once all safety settings are none)

Re: Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

#50

Post by moduspol »

sivers wrote: post #10

PAYMENT TANGENT for my fellow entrepreneurs here that take Visa/Mastercard payments: I tried to sign up to Lambda Labs just now to check out Hermes 3. Created an account, verified my email address, entered my billing info... ... but then it says they only accept CREDIT cards, NOT DEBIT cards. I had never heard of this, so I tried it anyway. I entered my business Mastercard (from mercury.com FWIW), that's never been r…

I think I read somewhere that forcing credit cards is a way for the merchant to completely avoid prepaid cards. Though obviously imperfect.

Privacy.com had to overhaul their card generation backend a few years ago specifically to handle merchants refusing their single-purpose card numbers due to them being detected as potentially prepaid cards. Though they did do it, so it might work for your case now.

I'm sure there's a fraud angle where someone signs up with a cheap prepaid card, runs up a huge bill, and then the business has no recourse. Though I'm not familiar with Lambda Labs or their billing.

Post reply on HN