Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras Inference: AI at Instant Speed

cerebras.ai

Post reply on HN
Print view View original

11–20 of 75 posts

Re: Cerebras Inference: AI at Instant Speed

#11

Post by throwawaymaths »

itishappy wrote: post #5
jedberg wrote: post #2

This is where we always assumed the industry was going. Expensive GPUs are great for training, but inference is getting so optimized that it will run on smaller and/or cheaper processors (per token).

Cerebras' Wafer Scale Engine is the opposite of small and cheap. https://cerebras.ai/product-chip/

That's totally nuts. How do they deal with the silicon warping around disabled cores or dark silicon? How long of hard running does it take before the chip gets fatally damaged and needs to be replaced in their system? Word on the street is that h100s fail surprisingly often, this can't be better

Re: Cerebras Inference: AI at Instant Speed

#13

Post by brylie »

It would be understandable that they are focused currently on inference speed, but features like structured output and prompt caching make it possible to build more capable LLM applications.

Does Cerebras support reliable structured output like the recent OpenAI 4o?

Re: Cerebras Inference: AI at Instant Speed

#17

Post by bigcat12345678 »

brylie wrote: post #13

It would be understandable that they are focused currently on inference speed, but features like structured output and prompt caching make it possible to build more capable LLM applications. Does Cerebras support reliable structured output like the recent OpenAI 4o?

This question is a bit out of context.

Cerebras is a startup producing innovative AI chips. Their chips are super cool, and I personally believe Cerebras is ahead of the industry and is on the right technical path. As a matter of fact, Cerebras started with HPC chips. Then pivoted to AI like everyone else.

They are still deep in the trench for survival.

Given that, they have very little software prowess compared to AMD (which has *terrible* software stack for AI GPUs look at https://github.com/ROCm/rdc, an equivalent to NVIDIA DCGM, which virtually has no maintainer, and no one is using it), NVIDIA (the golden standard of software stack for AI GPUs); and you are referring to structured output and prompt caching which are prominently developed by LLM research institutions (OpenAI Anthropic, each of which have way more funding than Cerebras)

In the end, educate yourself, and do not put unrealistic expectation on startups.

Re: Cerebras Inference: AI at Instant Speed

#19

Post by smusamashah »

Can really fast inference (e.g. 1M tok/sec) make LLMs more intelligent? I am imagining you could run multiple agents and can choose and discard outputs using other LLMs simultaneously. Will the output look more like a real thought process? Or will it remain just same?

Re: Cerebras Inference: AI at Instant Speed

#20

Post by glial »

smusamashah wrote: post #19

Can really fast inference (e.g. 1M tok/sec) make LLMs more intelligent? I am imagining you could run multiple agents and can choose and discard outputs using other LLMs simultaneously. Will the output look more like a real thought process? Or will it remain just same?

Many of the results in the 'agent' literature require several agents and many iterations to produce an output. See some examples here [1]. Getting these results in seconds instead of minutes or hours would be incredible - and would help with iteration and experimentation to improve algorithms.

[1] https://langchain-ai.github.io/langgraph/tutorials/multi_age...

Post reply on HN