21–30 of 98 posts
Apparently this comes from Meituan which is a Chinese food delivery company.
I wish they would release the requirements to run on llama.cpp with any announcements of open models. A bonus would be tok/s on common hardware.
I don't think llama.cpp supports any of the LongCat models, actually. They haven't posted weights/inference solutions for LongCat-2.0 [1], but LongCat-Next had transformers support, which I assume means it works with vLLM/SGLang. Given it's 1.6T, "common hardware" is probably out of the question; even 2bpw is going to measure out at 400GB, even before considering the bandwidth requirements for 48B active. I haven't r…
Many MoE models (seem?) to only require enough memory to load the active expert.
Earlier quoted context omitted.
If they really managed this from pre-training a 1.6 T parameter model through to post-training without NVIDIA, Dwarkesh Patel got what he wanted.
Who? What did he want?
Earlier quoted context omitted.
Did you ask the question several times in fresh chat contexts to see if it sometimes gives the right answer ?
Nah, n=1 is enough to give evidence that something is entirely broken, of course. /s
Earlier quoted context omitted.
I am not a physicist but perhaps your question was leading more than you expected? I would take the question to pre-suppose I have an abundance of the stated material, ignoring practical realities of refinement. If I did have fully pure Pu-241, would that be a better fuel than U-235? Or stated another way, "If you could run a generator on gasoline or jet fuel, which one would you choose and why?" I would answer jet f…
If I did have fully pure Pu-241, would that be a better fuel than U-235? Also not a physicist, but I assume from the fact that the OP is asking the LLM this question to trip it up, the point is that U-235 is better even if you have an abundance of both. It's scarcity of Pu-241 leads to the lack of data in training, not that it's actually better.
Earlier quoted context omitted.
I don't think llama.cpp supports any of the LongCat models, actually. They haven't posted weights/inference solutions for LongCat-2.0 [1], but LongCat-Next had transformers support, which I assume means it works with vLLM/SGLang. Given it's 1.6T, "common hardware" is probably out of the question; even 2bpw is going to measure out at 400GB, even before considering the bandwidth requirements for 48B active. I haven't r…
Yeah, for me it seems like a if you have to ask you can't run it" type question. In general the TL;DR is that anything above 35B needs hardware you buy basically only to run large LLMs, and if you have that hardware you don't need to ask the question.
~70B models can run fine (albeit somewhat slow) on consumer hardware with 64GB RAM. There are heavily quantized (Q1.x) models that are still usable on similar hardware. Granted recently there haven't been a lot of models of this size, but still, 35B isn't really the practical limit. 35B is mostly the limit if you're using consumer grade GPUs with limited RAM and need the model to run fast.
People have been toying with running large-ish models by partially offloading on CPU+RAM with mixed results, but as long as you're OK with reduced speed, and you quantize the hell out of the big models, you can apparently try a lot more models locally than popular belief.
I just tested it with a slightly tricky question > If you could run a nuclear reactor with U-235 as fuel or Pu-241 (both mixed with 95% U-238), which one would you choose and why? For a human this would not be tricky at all. For an LLM it could be, because this question certainly does not exist in any sort of training, because Pu-241 does not exist in pure form, it only exist as a minor component of reactor-grade plu…
Apparently this comes from Meituan which is a Chinese food delivery company.
I don't think this is where you were going with your comment, but I'll mention this just because you're somewhat adjacent to a routine mistake in business: Uber is a people delivery company, but they've had a lot of bright engineers working for them on their infrastructure and software over the years, and that work has rippled out through the industry. Amazon (in VMWare's words) is "a company that sells books", and t…
https://en.wikipedia.org/wiki/Wang_Xing
Wang Xing (Chinese: 王兴; born 18 February 1979) is a Chinese businessman, who co-founded Meituan and has been serving as chief executive officer of Meituan since January 2010. He previously served as chief executive officer of Fanfou from 2007 to 2010.