Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

351–360 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#351

Post by otabdeveloper4 »

adamtaylor_13 wrote: post #218
varispeed wrote: post #65

Earlier quoted context omitted.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.

> they'd fail on any novel problem not in their training data

Yes, and that's exactly what they do.

No, none of the problems you gave to the LLM while toying around with them are in any way novel.

Re: The path to ubiquitous AI (17k tokens/sec)

#353

Post by otabdeveloper4 »

smusamashah wrote: post #221
freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.

This doesn't work. The model outputs the most probable tokens. Running it again and asking for less probable tokens just results in the same but with more errors.

Re: The path to ubiquitous AI (17k tokens/sec)

#354

Post by otabdeveloper4 »

baalimago wrote: post #150

I've never gotten incorrect answers faster than this, wow! Jokes aside, it's very promising. For sure a lucrative market down the line, but definitely not for a model of size 8B. I think lower level intellect param amount is around 80B (but what do I know). Best of luck!

Make it for Qwen 2.5 and I'd buy it.

You don't actually need "frontier models" for Real Work (c).

(Summarization, classification and the rest of the usual NLP suspects.)

Re: The path to ubiquitous AI (17k tokens/sec)

#355

Post by ainch »

varispeed wrote: post #230
adamtaylor_13 wrote: post #218

Earlier quoted context omitted.

If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.

They don't solve novel problems. But if you have such strong belief, please give us examples.

Depends how precisely you define novel - I don't think LLMs are yet capable of posing and solving interesting problems, but they have been used to address known problems, and in doing so have contributed novel work. Examples include Erdos Problem #728[0] (Terence Tao said it was solved "more or less autonomously" by an LLM), IMO problems (Deepmind, OpenAI and Huang 2025), GPT-5.2 Pro contributing a conjecture in particle physics[1], systems like AlphaEvolve leveraging LLMs + evolutionary algorithms to generate new, faster algorithms for certain problems[2].

[0] https://mathstodon.xyz/@tao/115855840223258103

[1] https://huggingface.co/blog/dlouapre/gpt-single-minus-gluons

[2] https://deepmind.google/blog/alphaevolve-a-gemini-powered-co...

Re: The path to ubiquitous AI (17k tokens/sec)

#356

Post by jmalicki »

PhunkyPhil wrote: post #334
IanCal wrote: post #300

Earlier quoted context omitted.

The best coding model won’t be the best roleplay one which won’t be the best at tool use. It depends what you want to do in order to pick the best model.

I'm not saying you're wrong, but why is this the case? I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?

There is the pre-training, where you passively read stuff from the web.

From there you go to RL training, where humans are grading model responses, or the AI is writing code to try to pass tests and learning how to get the tests to pass, etc. The RL phase is pretty important because it's not passive, and it can focus on the weaker areas of the model too, so you can actually train on a larger dataset than the sum of recorded human knowledge.

Re: The path to ubiquitous AI (17k tokens/sec)

#357

Post by otabdeveloper4 »

robotpepi wrote: post #275
johnsimer wrote: post #249

Earlier quoted context omitted.

“ Models don’t get old as fast as they used to” ^^^ I think the opposite is true Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster

Are they becoming better at the same rate as before though?

No.

Re: The path to ubiquitous AI (17k tokens/sec)

#358

Post by soleveloper »

noveltyaccount wrote: post #261
soleveloper wrote: post #123

Earlier quoted context omitted.

In 20$ a die, they could sell Gameboy style cartridges for different models.

That would be very cool, get an upgraded model every couple of months. Maybe PCIe form factor.

Yes, and even holding couple of cartridges for different scenarios e.g image generation, coding, tts/stt, etc

Re: The path to ubiquitous AI (17k tokens/sec)

#359

Post by luyu_wu »

I think this is quite interesting for local AI applications. As this technology basically scales with parameter size, if there could be some ASIC for a QWen 0.5B or Google 0.3B model thrown onto a laptop motherboard it'd be very interesting.

Obviously not for any hard applications, but for significantly better autocorrect, local next word predictions, file indexing (tagging I suppose).

The efficiency of such a small model should theoretically be great!

Re: The path to ubiquitous AI (17k tokens/sec)

#360

Post by vessenes »

ttul wrote: post #273
rbanffy wrote: post #232

Earlier quoted context omitted.

> any factor of 10 being a new science / new product category, I often remind people two orders of quantitative change is a qualitative change. > The thing that I’m really very skeptical of is the 2 month turnaround. To get leading edge geometry turned around on arbitrary 2 month schedules is .. ambitious. Hopeful. We could use other words as well. The real product they have is automation. They figured out a way to c…

Having dabbled in VLSI in the early-2010s, half the battle is getting a manufacturing slot with TSMC. It’s a dark art with secret handshakes. This demonstrator chip is an enormous accomplishment.

Yeah and a team I’m not familiar with — I didn’t check bios but they don’t lead with ‘our team made this or that gpu for this or that bigco’.

The design ip at 6nm is still tough; I feel like this team must have at least one real genius and some incredibly good support at tsmc. Or they’ve been waiting a year for a slot :)

Post reply on HN