The long self talk process involved in figuring out how to use brackets to fix the math answer doesn't feel like reasoning to me. It feels like taking multiple shots in the dark and not learning anything from the last one, always returnning to square one
QwQ: Alibaba's O1-like reasoning LLM
281–290 of 435 posts
Re: QwQ: Alibaba's O1-like reasoning LLM
#282Is o1 even that good? It's doesn't even rank first on LMArena..
Re: QwQ: Alibaba's O1-like reasoning LLM
#283Is o1 even that good? It's doesn't even rank first on LMArena..
For very specific tasks, yeah it's good. For my usual coding tasks...no way.
Re: QwQ: Alibaba's O1-like reasoning LLM
#284We are lucky that Alibaba, Meta and Mistral sees some strategic value in public releases. If we it was just one of them, it would be a fragile situation for downstream startups. And they’re even situated in three different countries.
Re: QwQ: Alibaba's O1-like reasoning LLM
#285Is Alibaba's LLM the "Chinese LLM"?
It would appear to have been a U.S.-only game until now. As Eric Schmidt said in the YouTube lecture (that keeps getting pulled down), LLM's have been a rich-companies game.
Re: QwQ: Alibaba's O1-like reasoning LLM
#286Is Alibaba's LLM the "Chinese LLM"? It would appear to have been a U.S.-only game until now. As Eric Schmidt said in the YouTube lecture (that keeps getting pulled down), LLM's have been a rich-companies game.
Deepmind is in the UK and Mistral is in France?
Alibaba has been pumping out a bunch of useful models for a long time.
Re: QwQ: Alibaba's O1-like reasoning LLM
#287Is o1 even that good? It's doesn't even rank first on LMArena..
I haven’t been super impressed with it, and haven’t encountered any practical tasks I wanted to solve with an LLM where o1 worked any better than prompting 4o or Sonnet to use more extensive CoT.
There might be some narrow band of practical problems in between what other LLMs can do and what o1 can’t, but I don’t think that really matters for most use cases, especially given how much slower it is.
Day to day, you just don’t really want to prompt a model near the limits of its capabilities, because success quickly becomes a coin flip. So if a model needs five times as long to work, it needs to dramatically expand the range of problems that can be solved reliably.
Re: QwQ: Alibaba's O1-like reasoning LLM
#288Is o1 even that good? It's doesn't even rank first on LMArena..
My understanding was that the metric for LMArena is that one answer is “better” than another, for a deliberately 100% subjective definition of better.
My experience has been that typical LLMs will have more “preamble” to what they say, easing the reader (and priming themselves autoregressively) into answers with some relevant introduction of the subject, sometimes justifying the rationale and implications behind things. But for o1, that transient period and the underlying reasoning behind things is part of OpenAI’s special sauce, and they deliberately and aggressively take steps to hide it from users.
o1 will get correct answers to hard problems more often than other models (look at the math/coding/hard subsections on the leaderboard, where anecdotal experiences aside, it is #1), and there’s a strong correlation between correctness and a high score in those domains because getting code or math “right” matters more than the justification or explanation. But in more general domains where there isn’t necessarily an objective right or wrong, I know the vibe matters a lot more to me, and that’s something o1 struggles with.
Re: QwQ: Alibaba's O1-like reasoning LLM
#289Is Alibaba's LLM the "Chinese LLM"? It would appear to have been a U.S.-only game until now. As Eric Schmidt said in the YouTube lecture (that keeps getting pulled down), LLM's have been a rich-companies game.
you only think that because you haven’t been paying close attention
qwen, deepseek, yi - there have been a number of high quality, open chinese competitors
Re: QwQ: Alibaba's O1-like reasoning LLM
#290Is o1 even that good? It's doesn't even rank first on LMArena..
yes, it’s extremely good.
don’t overindex on the lmsys arena, the median evaluator is kinda mid