481–490 of 710 posts
Earlier quoted context omitted.
Did you try using the the talaas chat? Something stupid like 18k tokens/second. Think it's called Askjimmy or similar.
What is the model they're using there though? Interrogated, it claims it's a BERT variant and has capabilities around GPT-3 and below GPT-4. (Not that I believe it, it writes too well for GPT-3.) Hosted frontier models from two years ago would be much faster today, too.
Earlier quoted context omitted.
Your examples worked on phones for over a decade. Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.
It barely works even today, like Siri is laughably bad.
What Siri is missing is more logical solutions and answers for recipes, etc (still suck even with chatgpt integration).
I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
Completely fixed-function HW can't be used for training, it's inherently a statement that "this model is Good Enough and we are now gonna start just extracting its value instead of extending it". So yeah it's an inference moat but it's not a growth moat.
Makes perfect sense for a company trying to get into the compute business, not companies who wanna be in the creating-ASI business.
Still, I guess/hope they have teams doing it in-house anyway. Just not something they'd wanna make a huge amount of noise about, it doesn't look good for To The Moon valuations.
[0] https://www.dwarkesh.com/p/why-compute-might-get-10x-more-ex...
Earlier quoted context omitted.
How ? Do LLMs actually "know' when they don't "know" ?
They do this all the time, I'm using ChatGPT in Instant mode and it auto updates to thinking if my question is complex. Most of the time this works. To answer your question: A large language model itself does not know this (afaik). But chatbots are not "just LLMs" but a whole bunch of systems (and models) around them.
My understanding is that the chip carries the neural network itself while the weights are loaded on top, so any model that uses that same network topology should be loadable on demand.
Earlier quoted context omitted.
I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.
With thousands of token per second output it would be an enormous waste of resources. Such chips are clearly made to process thousands of conversations simultaneously. Not necessarily in parallel. All LLM workflows are turn based right now, there are often seconds between turns until tool calls finish or users type the next message. If the LLM response only takes a few milliseconds, the chip can process hundreds of o…
Sounds a lot like "640Kb ought to be enough for anybody"
Earlier quoted context omitted.
Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…
I tried it. I asked where Bruce Lee was born. It stated he was born in Hong Kong. I challenged it and it went further naming a hospital there. I stated he was born in San Francisco and it apologized and then said his father was a missionary traveling in America, which was also wrong. Bruce’s father was a famous Cantonese Opera singer and actor. This model had zero information right, while being fast in responding. Un…
I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}.
Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Generated in 0.024s - 14,163 tok/s", the page says). Slightly more powerful model while keeping this under a second, and this could easily become a qualitatively different form of autocomplete/text suggestion. Running in the background every couple keystrokes, or every time user stops typing for more than 500ms.
Earlier quoted context omitted.
It failed on my usual test. But it failed really fast: "A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage…
This farmer needs a tote.
Earlier quoted context omitted.
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
I think the lifecycle for these chips could stretch far longer. If you're offering these models on a two year lifecycle, then you'd be able to stand up your top tier (wouldn't need to be frontier) at high speed. Run (for example) Kimi K3 on it and give it a brand name: AcmeAI Carbon Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely…
This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints.
___
Specification HC1
Model Llama 3.1 8B (hardwired)
Process TSMC 6nm
Die size 815mm²
___
So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.