331–340 of 435 posts
32B is a good choice of size, as it allows running on a 24GB consumer card at ~4 bpw (RTX 3090/4090) while using most of the VRAM. Unlike llama 3.1, which had 8b, 70B (much too big to fit), and 405B.
what do you mean? I can easily run 70b on my macbook. Fits easily.
Does your MacBook really have a 24GB VRAM consumer (GPU) card?
We are lucky that Alibaba, Meta and Mistral sees some strategic value in public releases. If we it was just one of them, it would be a fragile situation for downstream startups. And they’re even situated in three different countries.
Now if we only can get Meta to understand what "Open Source" means so the word doesn't lose all meaning in the future.
Earlier quoted context omitted.
If your job or hobby in any way likes LLMs, and you like to "Work Anywhere", it's hard not to justify the MBP Max (e.g. M3 Max, now M4 Max) with 128GB. You can run more than you'd think, faster than you'd think. See also Hugging Face's MLX community: https://huggingface.co/mlx-community QwQ 32B is featured: https://huggingface.co/collections/mlx-community/qwq-32b-pre... If you want a traditional GUI, LM Studio beta 0…
4699$US. Quickest justification I ever made not to buy something
Earlier quoted context omitted.
what do you mean? I can easily run 70b on my macbook. Fits easily.
> what do you mean? I can easily run 70b on my macbook. Fits easily Does your MacBook really have a 24GB VRAM consumer (GPU) card?
Earlier quoted context omitted.
what do you mean? I can easily run 70b on my macbook. Fits easily.
Macs that can run it are quite a bit more expensive than a 3090. GPUs can also do finetuning and run other models with larger batch sizes which Macs would struggle with. Also, for the models that fit both, an nvidia card can run it much faster.
3090s aren’t sold anymore, so you have to scrounge for them in the used market, 4090s are a couple K? The local LLM scene is still pricey and confusing.
Earlier quoted context omitted.
I haven’t been super impressed with it, and haven’t encountered any practical tasks I wanted to solve with an LLM where o1 worked any better than prompting 4o or Sonnet to use more extensive CoT. There might be some narrow band of practical problems in between what other LLMs can do and what o1 can’t, but I don’t think that really matters for most use cases, especially given how much slower it is. Day to day, you jus…
I think the true edge of CoT models will come from layman usability. While I can easily prompt Claude for examples and then manually modify the code to fill in the gaps, general domain knowledge and technical understanding is absolutely required from the human sitting in front of the screen. With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make i…
I just punched that prompt into Sonnet 3.5 and o1 and I wouldn't say that o1 is doing anything better than Sonnet. o1 certainly didn't "do it", it gave me a very broad outline of how to accomplish that, from "Define requirements" to "Test and deply on Vercel"
So western controls on training hardware (hello NVIDIA) seem to have failed. I wonder if there will be any repercussions here.
So western controls on training hardware (hello NVIDIA) seem to have failed. I wonder if there will be any repercussions here.
Most of the papers in machine learning are coming from China. The vast majority. Most of the open source models on GitHub, too. Hailuo, Kling, Vidu, and Hunyuan (posted on Banodoko) blow Sora and Runway out of the water. China is dominating at this field. And if they begin releasing weights as open source, that'll mean foundation model companies can only bank on the thin facade of product. That's a really good strate…
Anyone actually reading papers in this area knows there's a lot of interesting work coming out of Chinese universities.
For many years, American immigration policy has meant a PhD is particularly beneficial to immigrants, with predictable effects on who earns PhDs [1]. It's no surprise that, when industry needs advanced degrees and people who can read and write papers, we find author lists full of one-syllable surnames.
[1] https://cra.org/wp-content/uploads/2024/05/2023-CRA-Taulbee-...
Earlier quoted context omitted.
what do you mean? I can easily run 70b on my macbook. Fits easily.
Macs that can run it are quite a bit more expensive than a 3090. GPUs can also do finetuning and run other models with larger batch sizes which Macs would struggle with. Also, for the models that fit both, an nvidia card can run it much faster.
Meanwhile 4090 are close to 2000CHF.
I have no doubt where the actual value is.