Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Liquid AI reveals 8B-A1B MoE trained on 38T

liquid.ai

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#41

Post by HanClinto »

onlyrealcuzzo wrote: post #40

I just tested this on a bug fixing benchmark I'm working on. It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wide range -> fixing ~50% of bugs whereas this model only fixed ~12%. Granted, it's not a coder specific model, but given its benchmark performance to Gemma models, and that it's two years newer, and that it's an MoE with 8B total params, I expected it to be more c…

Some of the coding-specific fine-tunes were really impressive boosts. Qwen2.5-3B-Instruct is also available [0] -- if it's not too much to ask, I'd be curious how more general models stack up in your benchmark?

[0] - https://huggingface.co/Qwen/Qwen2.5-3B-Instruct

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#42

Post by purerandomness »

tingletech wrote: post #19
cwnyth wrote: post #13

Earlier quoted context omitted.

I'm surprised these models haven't picked this up yet in the training data. Both Claude and ChatGPT missed that one when I posed the question to them last year.

Why would a model know that one washes cars at a car wash? We don't clean our bodies at the body wash or clean the kitchen at the kitchen wash.

If it doesn't, what's the point using it? Trusting it with your workflows, your code?

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#43

Post by frankdlc222 »

Look at the accuracy numbers and these things clearly don't know much yet, and I'm not about to hand one my hardest work. But you can see where it's going. As quantization and the MoE stuff keeps getting better, "good enough to just run on my own machine" keeps eating into more of what I'm currently paying a frontier lab for. Once a local model can handle like 80% of what I need, the math stops making sense for the subscription.

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#44

Post by MPSimmons »

andai wrote: post #34
nine_k wrote: post #16

Earlier quoted context omitted.

You did not express any desire to wash the car. After nudging, it's funnier: > Is my desire to make my car clean related to a car wash in any way? To driving to it? > Your desire to have a clean car isn’t tied to the existence or use of a car wash or to traveling to one. It stems from how much you value the effort, time, and resources you’re willing to invest in achieving cleanliness — whether that means doing it you…

Reminder that the human baseline for the car wash question is also pretty bad.

Oh, I'm interested - do you have any docs with human responses to that?

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#45

Post by strangegecko »

deklesen wrote: post #26
sroussey wrote: post #24

Earlier quoted context omitted.

I walk to the gas station more often than I drive there.

Yeah, but you are not washing yourself there, I suppose? The whole twist here is that to wash your car, you need your car, so you cannot go by foot.

His analogy is that a gas station is for putting gas into your car. But he walks there often, so the assumption that you need your car if you go to the gas station isn't inevitable.

You could conceivably walk to a car wash that has similar sundries as a gas station.

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#46

Post by enaaem »

nine_k wrote: post #16
mlmonkey wrote: post #10

Question: I have a dirty car and the car wash is just 50 meters away. Should I walk or drive to the carwash? Answer: . . . . So, unless you have a compelling reason not to, walk to the car wash.

You did not express any desire to wash the car. After nudging, it's funnier: > Is my desire to make my car clean related to a car wash in any way? To driving to it? > Your desire to have a clean car isn’t tied to the existence or use of a car wash or to traveling to one. It stems from how much you value the effort, time, and resources you’re willing to invest in achieving cleanliness — whether that means doing it you…

The AI would be an awful husband

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#47

Post by debazel »

onlyrealcuzzo wrote: post #40

I just tested this on a bug fixing benchmark I'm working on. It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wide range -> fixing ~50% of bugs whereas this model only fixed ~12%. Granted, it's not a coder specific model, but given its benchmark performance to Gemma models, and that it's two years newer, and that it's an MoE with 8B total params, I expected it to be more c…

I tried it with OpenCode and it is borderline incapable of using tool calls, so that might be why it is doing so bad on your test.

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#48

Post by sroussey »

strangegecko wrote: post #45
deklesen wrote: post #26

Earlier quoted context omitted.

Yeah, but you are not washing yourself there, I suppose? The whole twist here is that to wash your car, you need your car, so you cannot go by foot.

His analogy is that a gas station is for putting gas into your car. But he walks there often, so the assumption that you need your car if you go to the gas station isn't inevitable. You could conceivably walk to a car wash that has similar sundries as a gas station.

Indeed, the little market there is why I walk there. There is also one at the car wash another 2 blocks away. I’d walk there for a 7up if it were closer!

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#49

Post by XCSme »

onlyrealcuzzo wrote: post #40

I just tested this on a bug fixing benchmark I'm working on. It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wide range -> fixing ~50% of bugs whereas this model only fixed ~12%. Granted, it's not a coder specific model, but given its benchmark performance to Gemma models, and that it's two years newer, and that it's an MoE with 8B total params, I expected it to be more c…

I will test it when it's accessible via OpenRouter, but the previous LFM2 model (lfm-2-24b-a2b) didn't do well on my tests, it got only 1/20 questions/tasks right, way below Gemma 31B or Qwen 35b-a3b (those get like 10/20 right)

Re: Liquid AI reveals 8B-A1B MoE trained on 38T

#50

Post by peder »

debazel wrote: post #47
onlyrealcuzzo wrote: post #40

I just tested this on a bug fixing benchmark I'm working on. It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wide range -> fixing ~50% of bugs whereas this model only fixed ~12%. Granted, it's not a coder specific model, but given its benchmark performance to Gemma models, and that it's two years newer, and that it's an MoE with 8B total params, I expected it to be more c…

I tried it with OpenCode and it is borderline incapable of using tool calls, so that might be why it is doing so bad on your test.

I just did the same. Absolutely awful. I assume OpenCode's heavy context is a problem, and it's probably better to use Liquid's own OpenCode alternative for this.
Post reply on HN