Live data from Hacker News

Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

vicuna.lmsys.org

111–120 of 185 posts

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#111
post #33

Q: We need you to draw seven red lines. All of them strictly perpendicular; some with green ink and some with transparent. Can you do that? A: I'm sorry, but I am a text-based AI model and I don't have the ability to physically draw lines or use different colors of ink. However, I can help you visualize this concept by describing the red lines and the different ink colors. A red line with green ink: Imagine a traditi…

lmao what?

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#112
post #33

Q: We need you to draw seven red lines. All of them strictly perpendicular; some with green ink and some with transparent. Can you do that? A: I'm sorry, but I am a text-based AI model and I don't have the ability to physically draw lines or use different colors of ink. However, I can help you visualize this concept by describing the red lines and the different ink colors. A red line with green ink: Imagine a traditi…

lmao what?

Parent is referring to this: https://m.youtube.com/watch?v=BKorP55Aqvg

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#113
post #98
post #40

Try a foreign language. It was total failure with Turkish. ChatGPT is about a million times better at it although it's worse than it is with English.

LLaMA doesn't contain any meaningful training data in Turkish

afaik its 20 most popular latin based languages? should include turkish

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#114
post #91

Earlier quoted context omitted.

> However, if this is true, it seems pretty cool if I can train a GPT4-like model for only $300. Not train, but fine-tune a model that Facebook spent millions of dollars training.

They absolutely did not spend millions to train it. Credible estimates place the cost for an entity like Meta at about 30-100k, probably less since Meta likely owns the 256 A100x8s needed to train it. Even as an individual, it wouldn't cost you anywhere near a million if you only trained 13B and took advantage of volume pricing.

Could I get a source for that? Not that I don't believe you, but my napkin math puts the cost of training the 65b parameter model alone at a lot higher than 100k.

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#115
I can't understand how something called "a language model" can evaluate some input text. ChatGPT4 seems like it understands the text of the first assistance and properly assesses that it wasn't actually a description of the trip.

Similarly, the other day I asked it to explain a hard-to-understand philosophy quote and it explained it properly (I believe).

Can someone explain how they do this? It seems magical.

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#116

Earlier quoted context omitted.

“Can you name all of the soccer players whose names contain a Z”

It's so weird. No matter how I ask, it gets something wrong about this question. > Can you name 20 soccer players whose names contain a Z, and for each one, place brackets around the letter Z (Z)inedine (Z)idane Cristiano Ronaldo dos Santos A(ve)iro Andrej W(sz)ołek Nemanja Mati(c) Dusan Tadi(c) Gr(az)iano Pellè Jakub Bła(śc)z(ykowski) Kamil Glik Luk(asz) Pi(sz)czek Blai(se) Matuidi Miralem Pjani(c) Marek Ham(śi)k Lu…

There might not be a "Z" token for some of these names. A made-up example is "Lukasz" might tokenize to ["Luk", "asz"], so the model doesn't have any notion of how words are actually spelled. I suspect that if the body of training data came with some instructions on spelling it would know how to do this better, but it seems unlikely that there would be a natural language description of how Polish (?) names are spelled in the training data.

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#117

I can't understand how something called "a language model" can evaluate some input text. ChatGPT4 seems like it understands the text of the first assistance and properly assesses that it wasn't actually a description of the trip. Similarly, the other day I asked it to explain a hard-to-understand philosophy quote and it explained it properly (I believe). Can someone explain how they do this? It seems magical.

Either it's a lifeless recombination of text from the Internet, or an emergent understanding achieved through working with trillions of text tokens, despite lacking access to the full range of the human sensorium informing that text.

The jury is still out.

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#118
> 90%*

> * According to a fun and non-scientific evaluation with GPT-4. Further rigorous evaluation is needed.

Come on... I love any effort related to LLM, but it's really disingenuous to claim high quality in the title and then immediately discredit it after click-through.

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#119

> 90%* > * According to a fun and non-scientific evaluation with GPT-4. Further rigorous evaluation is needed. Come on... I love any effort related to LLM, but it's really disingenuous to claim high quality in the title and then immediately discredit it after click-through.

but it is indeed difficult to eval chatbots and LLM esp. considering most of them have actually seen the Internet data at least once.

Re: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality

#120

The work being done with Llama and Alpaca and so on is what I've found to be most interesting in the past few weeks. The fact that we are capable of producing such high quality conversational models that we can actually get to freely experiment with is really encouraging. I'm particularly excited about applications such as viper[1] and other forms of plugins as well as model ensembles. [1] https://viper.cs.columbia.e…

Turns out LLMs are not so inaccessible for individual users.

Exactly. but if you want some commercial use, still lacking good quality models.
Post reply on HN