Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing System One Models and Jev

typesafe.ai

Post reply on HN
Print view View original

161–170 of 513 posts

Re: Introducing System One Models and Jev

#161

Post by zenlikethat »

dinobones wrote: post #131

This is a good product but the naming/branding is pretty unfortunate. Typesafe.AI sounds like some typescript/structured output type of tool… What even is “system one” ? IMO the product/tech is really there, just needs better communication.

The model can't reason comprehensively (e.g., like Sol XHigh would to solve a complicated problem), but it's designed to be able to answer anything a human reasonably could quickly and intuitively, i.e., system one thinking: https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow

Re: Introducing System One Models and Jev

#162

Post by zmmmmm »

The eval is baffling me

> we assume there is a correct compute graph (a “workflow” represented in code) and use the predictions of the largest, smartest, and most expensive external models as reference probabilities. ... Rephrased: every model gets the same workflow. We test how they compare to the average of the smartest models (in this case, Astra and Fable).

They assume there is a correct graph, but they don't compare to that, they compare to the average of the smarts models? So the smartest models are getting it wrong but you compare that anyway as a benchmark? So the outcome is "how much of a Fable am I getting" etc. Why not compare the actually correct thing?

But then even on this hand constructed eval, the first plot is showing Jev at less than Sonnet 5 accuracy. It is barely better than Luna. There are two Opus 5's and two Sonnet 5's without explanation. What is the plot showing?

I gave up.

Re: Introducing System One Models and Jev

#164

Post by zenlikethat »

entrep wrote: post #68

This puts the human even more out of the loop I'll guess?

That's kinda the goal. Imagine all the automation in the world being able to embed intelligence directly inside it - factories could route based on more complicated questions, hardware could anticipate your needs. Customer support could be done without humans 90% of the time.

Re: Introducing System One Models and Jev

#166

Post by Mentlo »

Hm, would be good to understand the architecture better. Is this answering just from a world model informed prior? How informed is it by the information in the prompt? I can't see this maintaining calibration across all domains and all types of structured output.

Is there anything published on how it maintains calibration? Or when you say "outputs calibrated probabilities" you mean "as calibrated as frontier LLM models, just cheaper" - which is a different claim; as LLM's aren't particularly well calibrated

Re: Introducing System One Models and Jev

#167

Post by zenlikethat »

seinecle wrote: post #88

Can this be used in practice to write code?

Not in the traditional sense of a coding agent, but we think there's a ton of opportunity in using it for context management ("do we _really_ need to pass all these tokens to the agent?"), semantic linting ("how does this score against this AGENTS.md: "), etc.

Re: Introducing System One Models and Jev

#168

Post by sroussey »

WhitneyLand wrote: post #149
janalsncm wrote: post #113

Earlier quoted context omitted.

I don’t think it’s misleading if you compare on the use cases they suggested. It’s faster and cheaper (no idea if higher quality), so it’s immediately interesting for certain things. And if you buy their RLCD claims, this might be even better than huge models that know a bunch of irrelevant things.

What was misleading was the original title: "Jev: New frontier model 40-400x cheaper and 20-200x faster" I'm not the gatekeeper of who gets to call themselves a frontier model, but I don't think most people would count Jev in that group. It sounds false. If their specific claims hold up, then it would make more sense to say something like: "Advanced the speed/cost frontier for structured decisions"

I dunno, I would consider Waymo and Tesla to have frontier models.

I think AlphaFold and related are also frontier models.

Being an LLM does not seem like the qualifier for frontier.

Re: Introducing System One Models and Jev

#169

Post by potatoman22 »

Flere-Imsaho wrote: post #102
CompleteSkeptic wrote: post #65

Earlier quoted context omitted.

CEO here - that is right! I do agree that the comparison to LLM tokens is hard to understand (also because output tokens are not comparable). But yes, text or structured state (like a JSON with multiple pieces of text in) -> decisions out (e.g. choice maps to "match" statement, "score" maps to sorting, "noul" short for bernoulli maps to if-statements)

Hi - first congratulations, System One looks really promising. The Doom demo really help me, at least, to understand how System One differs from LLMs. However the first demo (Side-by-side demonstration) - I'm struggling to understand what is going on here!

I think that's to demonstrate its speed

Re: Introducing System One Models and Jev

#170

Post by alphazard »

There's a whole lot of information on this page that doesn't tell me anything about what this actually is. Can anyone spell out what the architecture is here?

They claim it's not an LLM, which I read as "not an auto-regressive token generator". I assume they are still using a transformer, otherwise they would be talking about the thing that's not a transformer, instead of all the fluff on the linked page. But they emphasize parallel generation, so is it like a text diffusion model?

Post reply on HN