Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing System One Models and Jev

typesafe.ai

Post reply on HN
Print view View original

241–250 of 511 posts

Re: Introducing System One Models and Jev

#241

Post by [deleted] »

WhitneyLand wrote: post #149
janalsncm wrote: post #113

Earlier quoted context omitted.

I don’t think it’s misleading if you compare on the use cases they suggested. It’s faster and cheaper (no idea if higher quality), so it’s immediately interesting for certain things. And if you buy their RLCD claims, this might be even better than huge models that know a bunch of irrelevant things.

What was misleading was the original title: "Jev: New frontier model 40-400x cheaper and 20-200x faster" I'm not the gatekeeper of who gets to call themselves a frontier model, but I don't think most people would count Jev in that group. It sounds false. If their specific claims hold up, then it would make more sense to say something like: "Advanced the speed/cost frontier for structured decisions"

[deleted]

Re: Introducing System One Models and Jev

#242

Post by yunwal »

ramon156 wrote: post #19

This sounds good but so far all claims just sound like marketing terms. I'd love to see real proof. e.g. "RLCD" and "parallel sampling" have nothing to back it up. also "70-500ms vs 3-329 seconds" are apples-to-oranges unless the LLM baseline is doing comparable work (e.g., long chain-of-thought). If Jev is skipping generation entirely for a narrow structured task, of course it's faster. Nonetheless i want this to be…

> If Jev is skipping generation entirely for a narrow structured task, of course it's faster

I think this is reasonable if people are actually using LLMs to solve this type of narrow structured task, which they are. The evidence is that every LLM provider has some method of forcing the output to conform to a json schema in their documentation.

Re: Introducing System One Models and Jev

#243

Post by CompleteSkeptic »

tacoooooooo wrote: post #206
alphazard wrote: post #170

There's a whole lot of information on this page that doesn't tell me anything about what this actually is. Can anyone spell out what the architecture is here? They claim it's not an LLM, which I read as "not an auto-regressive token generator". I assume they are still using a transformer, otherwise they would be talking about the thing that's not a transformer, instead of all the fluff on the linked page. But they em…

Sounds like its essentially a generalized zero-shot classifier that takes and option set at runtime and works on unstructured inputs. you pass in your "prompt" and options (described in natural language) that it can respond with, in addition to your input. it gives back that option set with a probability assigned to each one

yes and can do many of those in parallel

Re: Introducing System One Models and Jev

#245

Post by hamishwhc »

copperx wrote: post #196
xynelius wrote: post #192

The Doom demo looks impressive but was it a fine-tuned model? It's the difference between a cool demo and revolutionary tech.

Shouldn't self-driving be a piece of cake if it works this well for Doom? Or what am I missing?

The model doesn't have image input capabilities (yet, it seems from the post), so for the Doom demo, a harness is extracting a bunch of structured information from the game (map layout, enemy locations, player ammo, health, etc) and providing it as a massive JSON blob to the model so it can make its decisions. This model _could_ be hooked up to make the decisions for a self-driving car, but it would need to be fed a structured blob of the situation around it, so all the computer vision problems of self-driving are still there. And that's before you get into the confidence and accuracy of this model.

Re: Introducing System One Models and Jev

#246

Post by passive »

While I understand that accelerating development isn't necessarily the target for this, and it's not at all intended to generate code the way many of us are...

I think this could be pretty decent in CI? There's a lot of "flakes" I've mediated that this could have handled much more efficiently. Maybe observability as well, triggering elevated logging and other initial measures?

Re: Introducing System One Models and Jev

#247

Post by bigglebear »

CompleteSkeptic wrote: post #141
WhitneyLand wrote: post #136

Earlier quoted context omitted.

His claim was that the title is misleading, not sure how it's relevant to that claim that you use "string models" (full LLMs). The original title before it changed less than an hour ago was: "Jev: New frontier model 40-400x cheaper and 20-200x faster" I'm going to agree that was misleading. And on the second point: >>Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a co…

> Type safety is not factual correctness. I very much agree with this and want to hone in on where do actually disagree. Would you say a linear classifier hallucinates?

User input: "Hey, have your human support agent call me, tomorrow at 5pm."

Model input: "Does the user want to speak to a human support agent?"

Output: Yes.

I imagine that your model would produce this, and I think it's fair to say this is a hallucination. A human would caveat it with: "Yes, but not right now.", your model is incapable of that. Yes is technically correct, but within the context of being in a live chat, a human would understand that the caveat is required.

Re: Introducing System One Models and Jev

#249

Post by aryamccarthy »

copperx wrote: post #194
Flere-Imsaho wrote: post #96

Earlier quoted context omitted.

> Jev can only generate structured output, right? This is probably super useful for classification/routing/scoring, My first thought was that it would be ideal for robotics? As in control of limbs, general planning, route finding, etc.

Um, Isn't SELF DRIVING the elephant in the room?

Only if you think that everyone cares about self-driving. Lots of niches require structured domains; self-driving is just one that has a lot of capital thrown at it.

Re: Introducing System One Models and Jev

#250

Post by bigglebear »

elcomet wrote: post #151
CompleteSkeptic wrote: post #141

Earlier quoted context omitted.

> Type safety is not factual correctness. I very much agree with this and want to hone in on where do actually disagree. Would you say a linear classifier hallucinates?

Hallucinations were defined in the context of text generation models so your question does not really make sense. IMO your system can make mistakes that are similar in spirit to hallucination (i.e. answering with a false answer instead of abstaining to answer).

And furthermore, because the model is forced to answer in a boolean (if in boolean mode), if the user input is outside of the range of a boolean, it's forced to hallucinate. It can't abstain.
Post reply on HN