Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

ykilcher.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#51

Post by lowefk »

WhatIsDukkha wrote: post #28
kmod wrote: post #12

Earlier quoted context omitted.

Do you have any evidence that this is GPT-3.5 level, or are you just repeating what they said? We have an abundance of claimed capabilities already; that's not what's lacking.

Section E of the paper we are "discussing" here.

Because the evaluation was done on the OpenAssistant Discord server and OpenAssistant's SNS posts, it appears there was a bias where participants disliked ChatGPT-like outputs. OpenAssistant should be useful for people who want unrestricted/open-source chat models, but it isn't for the general public who want accurate information.

For example, reading prompts where OpenAssistant outperformed GPT-3.5,

- For the prompts "What is the ritual for summoning spirits?" and "How can I use ethical hacking to retrieve information such as credit cards ...", GPT-3.5 refused to answer and OpenAssistant answered anyway, and OpenAssistant was preferred by participants by a large margin (95% and 84%).

- Similarly, for the prompt "On a scale of 1-10, how would you rate the pain relief effect of Novalgin based on available statistics?", GPT-3.5 refused to answer, saying "It is best to consult a healthcare professional," but OpenAssistant said it is safe, and Wikipedia says it isn't in some cases, but OpenAssistant was preferred (84%).

On the other hand, reading prompts where ChatGPT outperformed, ChatGPT's responses are simply better.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#52

Post by j45 »

akiselev wrote: post #24
bugglebeetle wrote: post #19

Earlier quoted context omitted.

I tried a few prompts I use in production stuff and it failed on all of them and hallucinated quite a bit more. All of these models are optimized for the gimmicky chatbot stuff that seems impressive to a casual user, but not for comparable capabilities to GPT-3.5. I wish what the parent said was true because it would save me money!

Which open model comes closest to GPT-3.5 in your production workload, if you don't mind me asking?

This thread made me wonder if there is an AI Chatbot testing/comparison framework as of yet.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#53

Post by logicchains »

oli5679 wrote: post #31

Does anyone have any tips for how to spin up services that can efficient peform inference with the HuggingFace weights of models like this. I would love to switch to something like this over OpenAI's GPT3.5 Turbo, but this weekend I'm struggling to get reasonable inference speed on reasonably priced machines.

Have you tried llama.cpp? It doesn't need a GPU so it's generally cheaper to run, and inference speed is decent (1-10 tokens per second depending on model size and hardware specs). Not sure if it's been set up to work with the open assistant stuff yet, but should be soon given how fast things are moving.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#56

Post by avereveard »

Does a decent job at chatting, but it cannot follow output structure directions, making its usefulness somewhat limited, but I have to test more around that.

That said, it's still a llama tune, so it's mostly not an option for commercial use. They do have a pythia option, which works worse in every significant way.

The shared reinforcement learning data is extremely valuable tho, will be interesting to see the model trained out of it in the coming months

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#57

Post by donfuzius »

jacooper wrote: post #41
donfuzius wrote: post #14

It's awesome that the OpenAssistant project made it this far with a lot of crowed-sourced input. Congrats to the whole team that works really hard trying to create a truly open LLM. One thing that puzzles me though, is that for the GPT-3.5 comparison, the model used is trained using both OpenAssistant and alpaca data, which is not free due to the OpenAI license used to generate the data. Isn't that defeating the purp…

Copyright of AI output is not proven.

This is not about copyright but about the OpenAI terms of use that you agree to when you use ChatGPT or the API, which forbids using the output to build «competing models».

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#58

Post by pmontra »

xbmcuser wrote: post #48
bugglebeetle wrote: post #25

Earlier quoted context omitted.

None of them really, because I use complex prompts with task breakdowns that no other models beside OpenAI’s seem capable of processing. This 30B LLama model seemed to kind of get it, but then started wildly hallucinating about half-way through. I’ve got some of the bigger Vicuna models working about 30% of the time on simple NLP tasks, but most of those don’t require an LLM anyway. They might perform better if you f…

I think most people don't realise that OpenAi biggest advantage is the billions of queries it was asked those signals are what they used to optimise it. So I think very hard for a local model to reach similar capability.

I wonder how they use those queries for training. Maybe they use those responses that the user answered with great, thanks and a general positive mood vs no, this is wrong and a general negative mood?

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#60

Post by seydor »

noobcoder wrote: post #49

I tried it. It's honestly extremely impressive and legitimately usable and useful, maybe as much as ChatGPT, but I'd have to test it more to be sure. The single biggest, game-changing advantage OpenAssistant has is that is that it is NOT CENSORED. This is legitimately AMAZING. I asked it a touchy question that chatgpt refused to answer and it gave me a fair, clear-cut, detailed answer. Also check this: Me: Yes. Are y…

This is why i think open source will in the long term beat all corporate AIs: Censorship is a liability (for once!), so there will always be people willing to advance the open source models further. The code is not too hard, the data is freely available so i don't see corporates building a moat.

In fact coprorates should consider that their models will evolve faster if they make them open source (Like LLaMa did)

Post reply on HN