Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing System One Models and Jev

typesafe.ai

Post reply on HN
Print view View original

401–410 of 512 posts

Re: Introducing System One Models and Jev

#401

Post by adastra22 »

nylonstrung wrote: post #399
hdjrudni wrote: post #387

Earlier quoted context omitted.

> Assuming there's no funny business, the Doom demo is cool. The Doom demo seems very funny business. They're not feeding it video, they're feeding it a text description of what's going on in the game. It's not reading pixel data. I think LLMs would play a lot better with that input too but Jev does seem to have a huge speed advantage; I don't know if the other models could do that in real-time.

In a case like this it still seems more appropriate to encode that data in tabular form and use a tabular foundation model

Forgive my ignorance. Tabular foundation model?

Re: Introducing System One Models and Jev

#402

Post by postalcoder »

Gecko4072 wrote: post #350
postalcoder wrote: post #179

This has the potential to be huge for computer use. OpenAI has been teasing how fast computer use is with their models running on Cerebras chips but the difference here is a burning hole in your pocket.

Like which elements to select? Similar to the doom and wikipedia runs?

Yeah. Computer use is essentially a model navigating the OS-provided accessibility tree. I imagine a model trained on it would operate the computer exactly as we saw it control Doom.

https://developer.apple.com/library/archive/documentation/Ac...

Re: Introducing System One Models and Jev

#403

Post by adastra22 »

soleveloper wrote: post #358
jacobgold wrote: post #81

First, congrats to the team on launching something genuinely interesting and new. Seems like a more accurate title would be "Jev: Trading general purpose generation for fast typed inference" or something like that. This is interesting, but the speed comparison seems misleading? A generative model that can output code in a Turing-complete language can do anything a computer can do. Jev can only generate structured out…

I think the meaning of can't hallucinate in this model is that the type won't be hallucinated. So if the generated schema is for a tool call for calculator , then the numbers will be valid numbers for sure (and not random words). To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a…

AFAICT it is the same interface as you describe, but the underlying inference algorithm is fundamentally different, hence the speed gains. There is an application I am currently working on right now where this typed output predictor is the performance bottleneck. I'd be very interested to see how this performs.

Re: Introducing System One Models and Jev

#404

Post by vintermann »

> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle.

Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of rules for letters and letter combinations which may or may not result in the same sounds, hand-coded as a huge if tree by a linguist not a programmer)? What about their relatives, do they match by the same rules? Should we incorporate domain knowledge about local naming customs? Etc etc.

I pointed a coding agent to this problem, and it aggressively started coming up with complex scoring rules and testing them against real datasets. Which led to sort-of acceptable results, but it still missed lots of cases which were obvious to a human, and had false positives which were obvious to a human. Which I could trade off, and slightly improve, with more back and forth with the coding agent.

Pointing a good LLM to all the information about two people, would of course give great results. Maybe even better than human judgment. But I can't do that for 100000^2 people, it would be too expensive in all sorts of ways. I need a fast, reliable scorer. I could maybe train an embedding, but that would be a huge job and where would I get the quality data?

Re: Introducing System One Models and Jev

#405

Post by adastra22 »

8note wrote: post #218
dbbk wrote: post #100

Earlier quoted context omitted.

When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can disregard the result - that'd be different from hallucinating where it believes it's correct

if it puts a high confidence value on a wrong answer, thats still hallucinating, no? llm hallucinations are high probability tokens that are incorrect vs the real world

No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.

Re: Introducing System One Models and Jev

#406

Post by paraschopra »

Cool approach, i think less latency and cost is the way to go.

Here's how this would have likely been made.

- Tiny transformer or equivalent model (maybe a few bn or so?), explaining latency and cost

- Questions are sent in parallel to multiple copies of it (I'm sure they're edge located)

- The model is post-trained for calibration in a wide variety of data (the recipe is relatively simple, and likely targeted on distillation of logprobs / confidence of a bigger model)

Notice how cost is ONLY for input tokens as output is merely numbers (few tokens) because input could be huge (questions and options).

At 0.042-per-million price they have, Astra estimates the model to be 3bn parameters.

One could replicate this by post training Qwen 3.5 2Bn. I expect people to do so soon!

Re: Introducing System One Models and Jev

#409

Post by NitpickLawyer »

soleveloper wrote: post #358
jacobgold wrote: post #81

First, congrats to the team on launching something genuinely interesting and new. Seems like a more accurate title would be "Jev: Trading general purpose generation for fast typed inference" or something like that. This is interesting, but the speed comparison seems misleading? A generative model that can output code in a Turing-complete language can do anything a computer can do. Jev can only generate structured out…

I think the meaning of can't hallucinate in this model is that the type won't be hallucinated. So if the generated schema is for a tool call for calculator , then the numbers will be valid numbers for sure (and not random words). To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a…

Yeah, I thought about constrained generation as well. I've actually done something similar with local models before. And you can even get a "confidence" score by looking at the logits (something along the lines of logprob("YES") + logprob("Yes") + logprob("yes") - logprob("NO")...

There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.

That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.

edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...

Re: Introducing System One Models and Jev

#410

Post by throwaway314155 »

apexalpha wrote: post #381
qingcharles wrote: post #321

Earlier quoted context omitted.

My primary beef with smart home (having tried it) is that every person that visits your home ends up confused about some element of it. A light switch that goes up and down is universally understood.

My smarthome has regular switches and wifi. There's no reason to not do both. Also a quick NFC sticker in each room taking you to a small HTML site containing settings (temp, ventilation, lights, shutters, setting a alarm by the lights) has been golden. No one wants to: download Shelly app + AC app + look for ventilation IR controller + figure out how casting works for the TV + figure out how to use the Shelly app to…

That's clever and all - solid setup, good work. But I still think you either overestimate the average house guest or have particularly savvy/young house guests.
Post reply on HN