Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

lamini.ai

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#51

Post by anonzzzies »

charcircuit wrote: post #50
ksaj wrote: post #45

Earlier quoted context omitted.

This needs citation. These are not the same things. It will get numerical references right if it has sources used in the model, but it isn't doing any numerical calculations.

Just look at any papers that put models through mathematical benchmarks. The model isn't memorizing these problems. For example I just generated 2 random 64 bit integers and asked ChatGPT to add them. "6769545085823578960 + 16027170449476717488" ChatGPT said the answer is 22796715535300296448. It got the correct answer even though the problem wasn't in its training data.

Yep, as always, people (and LLMs) take stuff for granted because they read it somewhere months ago. That’s why we are doomed; everyone believes anything without question if it’s not against their personal agenda.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#52

Post by batch12 »

leobg wrote: post #40
batch12 wrote: post #32

I've been playing a bit with stacking transformer adapters to add knowledge to models and so far it has met my needs. It doesn't have the same illusion of intelligence, but so far it's just as good as a multitasking intern, so I am still having fun with it. I wonder if this is basically doing the same thing.

Interesting. Do you know if this can be done with Sentence Transformers, too? Picking a good performing one from HF. Then training an adapter for the domain (unsupervised). Then adding another one using actual training triplets (base, similar, non-similar)?

I haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers.

Check out https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA (https://arxiv.org/pdf/2106.09685.pdf) That should get you started.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#53

Post by vidarh »

ksaj wrote: post #45
charcircuit wrote: post #43

Earlier quoted context omitted.

Chat GPT does have counting logic. The math model is encoded inside of the language model.

This needs citation. These are not the same things. It will get numerical references right if it has sources used in the model, but it isn't doing any numerical calculations.

I can feed ChatGPT code that does calculations (and have) and have it calculate the right answers. It also gets it wrong a lot, so it's not good at that, but any notion that it can't do numerical calculations is easy to disprove.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#54

Post by vidarh »

avereveard wrote: post #9
primordialsoup wrote: post #8

Congrats! I went to your demo and asked for words that end in agi. This is what I got: -- agi, agi, agi, agi, agi, agi, agi These are some of the words that end in agi. You can also use the word agi in a sentence. For example, "I am going to the grocery store to get some agi." These are some of words that end in agi. These are some words that end in agi. maximize, maximize, maximize, maximize, maximize, maximize, max…

yeah as usual these model can barely sustain a conversation and fall apart the moment actual instructions are given. typical prompt they fail to udnerstand: "what is pistacchio? explain the question, not the answer." all these toy llm: "pistacchio is..." gpt is the only one that consistently understand these instructions: "The question "what is pistachio?" is asking for an explanation or description of the food item.…

It only makes them useless.of you insist on asking them in ways you already know will provide bad results instead of adapting your prompts.

This is a bit like complaining that your compiler refuses to produce the right outputs for code you've already determined is incorrect.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#55

Post by cassianoleal »

two_in_one wrote: post #46
brookst wrote: post #10

Earlier quoted context omitted.

It's a fair criticism, and ChatGPT does better, but this isn't a great test of model quality. All LLMS that rely on tokenization struggle with being introspective on language. Try asking chatGPT to count how many e's are in a sentence, or to list all words that start with "to" and end wide "de". I haven't heard anyone describe the phenomenon clearly, but I expect it is a challenge with reasoning over both intent of t…

well, here we go, ChatGPT in GPT-4 mode: There are 7 instances of the letter 'e' in the sentence: "Try asking chatGPT to count how many e's are in a sentence." another one: The words with the letter 'e' from the sentence "Try asking chatGPT to count how many e's are in a sentence" are: asking sentence and another, notice the last one: Here are some English words containing three instances of the letter 'e': Neverthel…

On the last list, the only word that does not comply with the constraints (having 3 'e's) is "Demeanor", which has only 2. Not great but also not as horrible as you make it sound.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#56

Post by avereveard »

vidarh wrote: post #54
avereveard wrote: post #9

Earlier quoted context omitted.

yeah as usual these model can barely sustain a conversation and fall apart the moment actual instructions are given. typical prompt they fail to udnerstand: "what is pistacchio? explain the question, not the answer." all these toy llm: "pistacchio is..." gpt is the only one that consistently understand these instructions: "The question "what is pistachio?" is asking for an explanation or description of the food item.…

It only makes them useless.of you insist on asking them in ways you already know will provide bad results instead of adapting your prompts. This is a bit like complaining that your compiler refuses to produce the right outputs for code you've already determined is incorrect.

Asking LLM from things they learned in training mostly result in hallucinations and in general makes you unable to detect by which amount they are hallucinating: these models are unable to reflect on their output, and average output token probability is a lousy proxy for confidence scoring their results.

On the other hand, no amount of prompt engineering seems to make these LLM able to do question and answer over source documents which is the only realistic way by which factual information can be retrieved

You're welcome to bring examples of it tho if you're so confident.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#57

Post by vidarh »

avereveard wrote: post #56
vidarh wrote: post #54

Earlier quoted context omitted.

It only makes them useless.of you insist on asking them in ways you already know will provide bad results instead of adapting your prompts. This is a bit like complaining that your compiler refuses to produce the right outputs for code you've already determined is incorrect.

Asking LLM from things they learned in training mostly result in hallucinations and in general makes you unable to detect by which amount they are hallucinating: these models are unable to reflect on their output, and average output token probability is a lousy proxy for confidence scoring their results. On the other hand, no amount of prompt engineering seems to make these LLM able to do question and answer over sou…

I've had ChatGPT build a fire nctioning website, write a DNA server, fill in significant portions of specs, all without the problems you describe. I'm never going back to doing things from scratch - it's saving me immense amounts of time every single day. The only reasonable conclusion is that the way you're promoting it is counterproductive.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#59

Post by avereveard »

vidarh wrote: post #57
avereveard wrote: post #56

Earlier quoted context omitted.

Asking LLM from things they learned in training mostly result in hallucinations and in general makes you unable to detect by which amount they are hallucinating: these models are unable to reflect on their output, and average output token probability is a lousy proxy for confidence scoring their results. On the other hand, no amount of prompt engineering seems to make these LLM able to do question and answer over sou…

I've had ChatGPT build a fire nctioning website, write a DNA server, fill in significant portions of specs, all without the problems you describe. I'm never going back to doing things from scratch - it's saving me immense amounts of time every single day. The only reasonable conclusion is that the way you're promoting it is counterproductive.

Good thing then that I specifically mentioned gpt as being able to follow instruction and that I was specifically mentioning the other models.

You're welcome to demonstrate the same ability on other models tho.

Re: Launch Lamini: The LLM Engine for Rapidly Customizing Models as Good as ChatGPT

#60

Post by leobg »

batch12 wrote: post #52
leobg wrote: post #40

Earlier quoted context omitted.

Interesting. Do you know if this can be done with Sentence Transformers, too? Picking a good performing one from HF. Then training an adapter for the domain (unsupervised). Then adding another one using actual training triplets (base, similar, non-similar)?

I haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers. Check out https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA ( https://arxiv.org/pdf/2106.09685.pdf ) That should get you started.

Thank you. Peft and adapters seem to be two different things though, no? AFAIK there are other libraries for adapters (forgot the name). Is peft what you were talking about when you said adapters in your original comment?
Post reply on HN