Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

291–300 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#291

Post by rbanffy »

VagabundoP wrote: post #252
rbanffy wrote: post #232

Earlier quoted context omitted.

> any factor of 10 being a new science / new product category, I often remind people two orders of quantitative change is a qualitative change. > The thing that I’m really very skeptical of is the 2 month turnaround. To get leading edge geometry turned around on arbitrary 2 month schedules is .. ambitious. Hopeful. We could use other words as well. The real product they have is automation. They figured out a way to c…

There might be a foodchain of lower order uses when they become "obsolete".

I think there will be a lot of space for sensorial models in robotics, as the laws of physics don't change much, and a light switch or automobile controls have remained stable and consistent over the last decades.

Re: The path to ubiquitous AI (17k tokens/sec)

#292

Post by rbanffy »

sowbug wrote: post #246
rbanffy wrote: post #186

Earlier quoted context omitted.

> I wonder if that applies? What's the big deal if a few parameter have a few bit flips? We get into the sci-fi territory where a machine achieves sentience because it has all the right manufacturing defects. Reminds me of this https://en.wikipedia.org/wiki/A_Logic_Named_Joe

Also see Adrian Thompson's Xilinx 6200 FPGA, programmed by a genetic algorithm that worked but exploited nuances unique to that specific physical chip, meaning the software couldn't be copied to another chip. https://news.ycombinator.com/item?id=43152877

I love that story.

Re: The path to ubiquitous AI (17k tokens/sec)

#293

Post by Windchaser »

freakynit wrote: post #175
codebje wrote: post #165

Earlier quoted context omitted.

RAM hoarding is, AFAICT, the moat.

lol... true that for now though

Yeah, just cause Cisco had a huge market lead on telecom in the late '90s, it doesn't mean they kept it.

(And people nowadays: "Who's Cisco?")

Re: The path to ubiquitous AI (17k tokens/sec)

#294

Post by 34679 »

Epskampie wrote: post #243
smusamashah wrote: post #221

Earlier quoted context omitted.

With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.

And then it's slow again to finally find a correct answer...

It won't find the correct answer. Garbage in, garbage out.

Re: The path to ubiquitous AI (17k tokens/sec)

#295

Post by jzymbaluk »

freakynit wrote: post #136
amelius wrote: post #128

Earlier quoted context omitted.

OK investors, time to pull out of OpenAI and move all your money to ChatJimmy.

A related argument I raised a few days back on HN: What's the moat with with these giant data-centers that are being built with 100's of billions of dollars on nvidia chips? If such chips can be built so easily, and offer this insane level of performance at 10x efficiency, then one thing is 100% sure: more such startups are coming... and with that, an entire new ecosystem.

You'd still need those giant data centers for training new frontier models. These Taalas chips, if they work, seem to do the job of inference well, but training will still require general purpose GPU compute

Re: The path to ubiquitous AI (17k tokens/sec)

#296

Post by FieryTransition »

audunw wrote: post #160
FieryTransition wrote: post #71

If it's not reprogrammable, it's just expensive glass. If you etch the bits into silicon, you then have to accommodate the bits by physical area, which is the transistor density for whatever modern process they use. This will give you a lower bound for the size of the wafers. This can give huge wafers for a very set model which is old by the time it is finalized. Etching generic functions used in ML and common fused…

Models don’t get old as fast as they used to. A lot of the improvements seem to go into making the models more efficient, or the infrastructure around the models. If newer models mainly compete on efficiency it means you can run older models for longer on more efficient hardware while staying competitive. If power costs are significantly lower, they can pay for themselves by the time they are outdated. It also means…

From my own experience, models are at the tipping point for being useful at prototypes in software, and those are very large frontier models not feasible to get down on wafers unless someone does something smart.

I really don't like the hallucination rate for most models but it is improving, so that is still far in the future.

What I could see though, is if the whole unit they made would be power efficient enough to run on a robotics platform for human computer interaction.

It makes sense they would try to make repurposing their tech as much as they could since making changes is frought with a long time frame and risk.

But if we look long term and pretend that they get it to work, they just need to stay afloat until better smaller models can be made with their technology, so it becomes a waiting game for investors and a risk assessment.

Re: The path to ubiquitous AI (17k tokens/sec)

#297

Post by FieryTransition »

robotpepi wrote: post #275
johnsimer wrote: post #249

Earlier quoted context omitted.

“ Models don’t get old as fast as they used to” ^^^ I think the opposite is true Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster

Are they becoming better at the same rate as before though?

In my unscientific experience, yes, but being better at a certain rate is hard to really quantify, unless you just pull some random benchmark numbers.

Re: The path to ubiquitous AI (17k tokens/sec)

#298

Post by Alifatisk »

empath75 wrote: post #225
jvidalv wrote: post #205

Earlier quoted context omitted.

Is super fast but also super inaccurate, I would say not even gpt-3 levels.

There are a lot of people here that are completely missing the point. What is it called where you look at a point of time and judge an idea without seemingly being able to imagine 5 seconds into the future.

“static evaluation”

Re: The path to ubiquitous AI (17k tokens/sec)

#299

Post by jtr1 »

The demo was so fast it highlighted a UX component of LLMs I hadn’t considered before: there’s such a thing as too fast, at least in the chatbot context. The demo answered with a page of text so fast I had to scroll up every time to see where it started. It completely broke the illusion of conversation where I can usually interrupt if we’re headed in the wrong direction. At least in some contexts, it may become useful to artificially slow down the delivery of output or somehow tune it to the reader’s speed based on how quickly they reply. TTS probably does this naturally, but for text based interactions, still a thing to think about.

Re: The path to ubiquitous AI (17k tokens/sec)

#300

Post by IanCal »

condiment wrote: post #271
Gareth321 wrote: post #158

Earlier quoted context omitted.

I think the next major innovation is going to be intelligent model routing. I've been exploring OpenClaw and OpenRouter, and there is a real lack of options to select the best model for the job and execute. The providers are trying to do that with their own models, but none of them offer everything to everyone at all times. I see a future with increasingly niche models being offered for all kinds of novel use cases.…

At 16k tokens/s why bother routing? We're talking about multiple orders of magnitude faster and cheaper execution. Abundance supports different strategies. One approach: Set a deadline for a response, send the turn to every AI that could possibly answer, and when the deadline arrives, cancel any request that hasn't yet completed. You know a priori which models have the highest quality in aggregate. Pick that one.

The best coding model won’t be the best roleplay one which won’t be the best at tool use. It depends what you want to do in order to pick the best model.
Post reply on HN