Does anyone know why they brand it an "inference chip"? Is it something at the hardware level that makes is unsuitable for training, or is it simply that the toolchain for training is massively more complicated to program?
Vsora Jotunn-8 5nm European inference chip
31–40 of 74 posts
Re: Vsora Jotunn-8 5nm European inference chip
#32I don’t get the negativity. The specs look impressive. It is always good to have competition. They announced tapeout in October with planned dev boards next year. Vaporware is when things don’t appear, not when they are on their way (it takes some time for hardware). It’s also strategically important for Europe to have its own supply. The current and last US administration have both threatened to limit supply of AI c…
>I don’t get the negativity.
Where do you see the negativity?
I don't believe labeling healthy skepticism and criticism as negativity to farm artificial sympathy in retaliation, does any good to anyone.
Humans have pattern recognition capabilities for a reason, and if a company is triggering that in them, then it's best expressed why(probably because they saw this MO before and got burned) instead of just cheerleading the unknown for fake positivity.
Re: Vsora Jotunn-8 5nm European inference chip
#33Does anyone know why they brand it an "inference chip"? Is it something at the hardware level that makes is unsuitable for training, or is it simply that the toolchain for training is massively more complicated to program?
Very simplified, AI workloads need compute and communications and compute dominates inference, while communications dominate training.
Most start-ups innovate on the compute side, whereas the techno needed for state of the art communications is not common, and very low-level: plenty of analog concerns. The domain is dominated by NVidia and Broadcom today.
This is why digital start-ups tend to focus on inference. They innovate on the pure digital part, which is compute, and tend to use off-the-shelf IPs for communications, so not a differentiator and likely below the leaders.
But in most cases coupling a computation engine marketed for inference with state of the art communications would (in theory) open the way for training too. It's just that doing both together is a very high barrier. It's more practical to start with compute, and if successful there use this to improve the comms part in a second stage. All the more because everyone expects inference to be the biggest market too. So AI start-ups focus on inference first.
Re: Vsora Jotunn-8 5nm European inference chip
#34I don’t get the negativity. The specs look impressive. It is always good to have competition. They announced tapeout in October with planned dev boards next year. Vaporware is when things don’t appear, not when they are on their way (it takes some time for hardware). It’s also strategically important for Europe to have its own supply. The current and last US administration have both threatened to limit supply of AI c…
It's not just competition.
These kinds of things-- cheaper-than-NVIDIA cards that can produce a lot of tokens or run large models cheaply are absolutely necessary to scale text models economically.
Without things like these-- those Euclyd things, those Groq things, etc. no one will be able to offer up big models at prices where people will actually use them, so lack of things like this actually cripples training of big models too.
If the price/token graph is right, this would mean 2.5x more tokens, which presumably means actually using multiple prompts to refine something before producing the output, or to otherwise produce really long non-output sequences during the preparation the output. This also fits really well with the Chinese progress in LLM RL for maths. I suspect all that stuff is totally general and can be applied to non-maths things too.
Re: Vsora Jotunn-8 5nm European inference chip
#35Impressive numbers on paper, but looking at their site, this feels dangerously close to vaporware. The bottleneck for inference right now isn't just raw FLOPS or even memory bandwidth—it's the compiler stack. The graveyard of AI hardware startups is filled with chips that beat NVIDIA on specs but couldn't run a standard PyTorch graph without segfaulting or requiring six months of manual kernel tuning. Until I see a d…
This 100x. I used to work for one of those startups. You need something crazy like a 10x performance advantage to get people to switch from Nvidia to some here-today-gone-tomorrow startup with a custom compiler framework that requires field engineer support to get anything to run.
The outcome is that most of custom chips end up not being sold on the open market; instead their manufacturers run them themselves and sell LLM-as-a-service. E.g. Cerebras, Samba Nova, and you could count Google's TPUs there too.
Re: Vsora Jotunn-8 5nm European inference chip
#36Does anyone know why they brand it an "inference chip"? Is it something at the hardware level that makes is unsuitable for training, or is it simply that the toolchain for training is massively more complicated to program?
Probably because their software only supports inference. It's relatively easy to do via ONNX. Training requires an order of magnitude more software work.
Re: Vsora Jotunn-8 5nm European inference chip
#37The fact that I have to give them an email for details just feels immediately like a B2B-scam.
Hope they can figure out software, but what im seeing isn't super-promising
Re: Vsora Jotunn-8 5nm European inference chip
#38I love that the JS loads so slow on first load that it just says "The magic number: 0 /tflops"
It loaded fine for me, but that slash before the unit was a bit smelly. :| Just a tiny edit, but it's a rather core part of their message so they should probably notice and format it correctly before publishing.
I think it could be intended, there is a SI document that says something like "x /unit" is a common way to indicate the unit of a quantity, which a guy I know is using as basis for advocating for that ugly display standard.
Re: Vsora Jotunn-8 5nm European inference chip
#39Earlier quoted context omitted.
Six months of one developer tuning the kernel? That seems like not much compared to the hundreds of billions of dollars US companies currently invest into their AI stack? OpenAI pays thousands of engineers and researchers full time.
more like 100 developers for 2 years
its the new "...and tell me if the picture has a bird"
Re: Vsora Jotunn-8 5nm European inference chip
#40Impressive numbers on paper, but looking at their site, this feels dangerously close to vaporware. The bottleneck for inference right now isn't just raw FLOPS or even memory bandwidth—it's the compiler stack. The graveyard of AI hardware startups is filled with chips that beat NVIDIA on specs but couldn't run a standard PyTorch graph without segfaulting or requiring six months of manual kernel tuning. Until I see a d…
Six months of one developer tuning the kernel? That seems like not much compared to the hundreds of billions of dollars US companies currently invest into their AI stack? OpenAI pays thousands of engineers and researchers full time.
It is. The problem is latency. All these fields are moving very fast, and so it doesn't sound bad spending 6 months tuning something, but in reality what is happening is that during those 6 months the guy who built the thing you're tuning has iterated 5 more times and what you started on 6 months ago is now much much better than what you got handed 6 months ago whilst simultaneously being much worse than what that person has in their hands today. If the field you're working in is relatively static, or your performance gap is large enough it makes sense. But in most fields the performance gap is large in absolutely terms but small in temporal terms. You could make something run 10x faster, but you can't build something that will run faster than what will be state of the art in 2 months.