241–250 of 297 posts
Earlier quoted context omitted.
> GLM 5.3 Flash at Q4, AA score 57 That AA score is for the original model only
Then take DeepSeek V4 flash with AA score 52. Runs unquantized on 2x DGX spark with 1M context.
Earlier quoted context omitted.
They already publish gpt-oss which is several generations better than gpt-3
GPT-3 is a different model than gpt-oss and is therefore not an answer to the question. I cannot stand using gpt-oss, but I miss some of the creative spark of GPT-3 davinci dearly.
GPT-3 is probably one of the last models trained without synthetic data. Meanwhile gpt-oss is rumored to be only trained on synthetic data.
I also miss the 2020 GPT-3 based AI Dungeon fine tune.
Earlier quoted context omitted.
When we consider: * LLM usage is new for the world * Models are evolving quickly with high worldwide competition * Hardware is evolving despite RAM shortages Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what y…
It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
Furthermore we continue to follow the path that image gen neural networks took. In that domain hardware requirements reached a peak and then started sharply declining to where we are today where a plain old video card can rapidly generate images that took a supercomputer not that long ago. So it's reasonable to assume that performance of such a system could potentially even increase over time.
Earlier quoted context omitted.
> What if the model is hopelessly obsolete, and thus no demand, but I want that specific model? You can still find a lot of old and completely outdated models on OpenRouter. The providers can scale serving of models up and down as demand arrives, so models don't generally disappear. They're just kept in the mix and the clouds will allocate hardware to it if someone is willing to pay. In the odd case that it disappear…
> There is a certain personality type that cannot tolerate any uncertainty and must lock everything in right now against all future possibilities. I thought HN banned personal attacks. I'm in this sentence and I don't like it. /s I just buy the good apple hardware because it's good, and it also happens to run local models. It's not as good for the dollar, don't get me wrong, but I'm not going to develop iOS without a…
Earlier quoted context omitted.
It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
I think that's overly pessimistic. Here's [1] a video of somebody running it on a ~$6000 rig and getting around 14T/s for complex prompts (about double that for simpler prompts). Payback time is going to depend on your electric cost/consumption. In most domains cloud providers end up charging a significant premium rather than a offering a scale enabled discount, relative to local at retail costs. That will almost cer…
That’s less than 164USD worth of GLM5.3 tokens on the inference market. So that 6000 USD rig will take 3 years to break even - and only if it runs continuously. And this is being generous, as it’s not even taking quantisation into account.
I previously posted that DS4Flash was _good_ but not _great_ on two DGX Sparks, but I have to say that GLM-5.3 is pretty amazing. It's been able to tackle all the random hard problems I've thrown at it and it has the intuition that DS4Flash seems to lack. We're nowhere near a Fable-class model IMO, but things are going to get interesting in this next year.
> We're nowhere near a Fable-class model IMO, but things are going to get interesting in this next year. I'm wondering of you could clarify your thoughts on this. I've had a hard time evaluating what Fable-class actually is capable of that sets them (or really it) apart from other models in a very significant way.
- Get an M5 Ultra with 512 GB of memory. - A couple of generations of improvements to the base model training. - A couple of generations of improvements to the architecture for running it as efficiently as possible. - A couple of generations of open coding agent harnesses like Pi and OpenCode.
And given how fast each generation comes and goes, we are now probably guaranteed superhuman coding assistant that requires 350-ish watts of power, is smaller than a toaster, and can code at 50 TPS. And even if the M5 cost is substantial, is WAY WAY cheaper than what I though would be even possible within a reasonable amount of time.
I cant think of anything in history that has improved at this rate... And yea, theres a lot of AI hype, but the amount of people that dont realize how insane this is, suprises me.
Earlier quoted context omitted.
When we consider: * LLM usage is new for the world * Models are evolving quickly with high worldwide competition * Hardware is evolving despite RAM shortages Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what y…
It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.