451–460 of 682 posts
Earlier quoted context omitted.
I would love to see any good research projects about it but i have the feeling that Frontier with MoE is making too fast of a progress so that a customized model would always be worse and that the MoE part is actually going somehow in this direction. On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant differ…
Capability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition. That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto,…
Meta is rocking AI. As of last week I have been using their excellent muse coding harness with their model Muse Spark 1.2. Starting this morning I am running their new local 30B model muse-glimmer on my old MacMini 32G using Ollama (remember to increase the context size!) and pi coding harness. I am getting good results with muse-glimmer running locally, with the caveat that everything runs slowly (e.g., give it a ta…
> Amazing! You have a total of 128.94 TFLOPS of computing power. 71.3% percentile on scale of "GPU Poor" to "GPU Rich"
The way I see it, these are amazing machines that the richest folks are hovering up. I think they should be in the hands of regular people as much as possible. They depend on an incredibly global, increasingly fragile supply chain. If the become impossible to produce, their value would increase tremendously. I think they will become really valuable to you to use the tokens directly, but if that isn't the case, they can be rented out or resold. Please don't just buy any hold. Let's try to get as many people that can use them for decent things that help humans. For example:
https://spectrum.ieee.org/small-language-models-ai-pharmaceu...
Still needs 32-64GB memory to run it locally. 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany. A more practical model would be a language specific (e.g Python or JVM language) and excellent at tool calling and reasoning. Maybe that way they can shrink it even more.
Coming from the PC games industry in the 90s and early 2000s, it was a struggle to run some of the games on release. 90%* of people wouldn't be able to play the AAA games on release (think Crysis, etc). This period of local LLMs reminds me of that time, whereby the hardware just isn't there yet. Give it time, and the prices will drop. * total guess
Some interesting findings from the chat template designs: 1. The template name is Onyx ATEM as found in the tool call exception message 2. It appears to be following a harmony-style chat template. But the tool use seems to be a xml like : / / 3. atem: a internal joke of meta in reverse? https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/mai...
also perhaps taking some small joy from the lexical similarity to aten[0] namespace that lies at the heart of pytorch
[0]: https://github.com/pytorch/pytorch/blob/main/aten/src/README...
Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single box overnight? That moment for LLMs is near. It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. And I think the data center buildout will end in car…
brains do it with 20 watts because theyre analog. llms require massive amounts of power and this isnt changing any time soon without a breakthrough
On the other side, advancement in quantum computers would make current LLM inference much faster. Because of the extreme cooling needed, i dont think the energy demand would become less.
With AI companies talking about AGI, i sometimes wonder if they really need the machines for serving inference to customers, or they have a formula for computational capacity that could run an AGI, and they just want to reach that level.
Earlier quoted context omitted.
I kick myself a couple times a week for not getting the 512GB Mac Studio in February. I was holding out for an M4 or M5 chip...
I was about a week away from buying a very tricked out MacBook Pro with 128 GB RAM, but was on vacation and worried about it arriving while I was away, and then the price hikes went into effect. Grumble. Oh, well. Serves me right.
The best time to plant a tree…