281–290 of 712 posts
Earlier quoted context omitted.
What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?
Reasoning models are the same speed. They’re just post trained with RL to do CoT inside tags like before a tag like There’s no difference in the inference implementation, parameter count, or speed.
But yeah, there are a lot of factors, so it's hard to answer, and tokens/s isn't the right question.
> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
The methods for splitting weights across multiple chips are well established. Groq/Cerebras can't hold a model on one chip either.
Earlier quoted context omitted.
> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least. What are some examples?
There's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience
Earlier quoted context omitted.
Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…
I'm not sure inference speed is always the slowest thing for me right now. The agent is running tests, loading webpages, etc, which all take time. I don't know if a fast agent would speed things up in all cases. That said, it obviously depends on the project.
A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.
Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
The demo: https://chatjimmy.ai/
Generated in 0.051s • 14,092 tok/s
Impressive...
Given gpt 5.5 was very good to me and gpt 5.6 series seems not boost too much, i kinda like the way bake the model weight to the chip, and connect multiple chip to serve the large scale model and allow respin some parts(ROM like?) to do model weight update, maybe this seems sustainable, the future is exciting
Earlier quoted context omitted.
There's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience
I wanted some examples they actually experienced. Because I use these things daily and haven’t seen a hallucination in a long long time.
Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.
Cerebras already runs large models like Kimi 2.6 or GLM at like 30x speed. 100 times is next year, not six years. You can actually test it out on their website, just imagine 3 x faster and maybe 15% smarter.
If things like this get traction, will we need all the datacenters?