[flagged]
Introducing System One Models and Jev
431–440 of 513 posts
Re: Introducing System One Models and Jev
#432Amazing work by the team! Looks like they've traded accuracy for speed and this is most likely going to be the case with the next class of models.
This is a valid tradeoff for one-off responses but if we're dealing with a distributed system (eg: Kafka), then only the high-confidence responses (>0.8) should move forward as input to the next service. If a low confidence output is propagated, then it can break the entire chain.
Re: Introducing System One Models and Jev
#433TLDR: Like an LLM, the input is a string, but the output is not a completion. The output is a ranking of elements from a certain enum (e.g, [Yes/No], [A/B/C/D]). They use a technique called Reinforcement Learning for Calibrated Decisions (RLCD) instead of RLHF. Also, inference is a lot faster.
https://docs.typesafe.ai/primitives has a much better explanation
Re: Introducing System One Models and Jev
#434"can't hallucinate" feels like some word game Olympics
Re: Introducing System One Models and Jev
#435Earlier quoted context omitted.
I mean, it's a structured output model that (apparently) can't hallucinate. I don't mind the name.
It can't hallucinate, but it doesn't mean it can't make wrong decisions. Just because it adheres to a specific output format at all time, while LLMs have the output format at their mercy, then the claim of not hallucinating is made technically true. I think that this specific part is not super interesting if your harness just recovers from invalid LLM outputs. The latency and cost - yes, those are super interesting.
You can get rigid output format from "classic" LLMs https://docs.vllm.ai/en/latest/features/structured_outputs/ though model support is limited.
Would like to have something like in the original post but open weights.
Re: Introducing System One Models and Jev
#436I was previously working on LLMs to extract key info from data rooms for energy assets, and this looks great for that use case.
"Does this contract contain ____?" is a pretty typical query for many industries, and then you can have follow up questions that nest down into further info about X, Y or Z thing.
Looks really good for that use case, especially with certainty as part of the output, as you could flag things that didn't have high enough of a certainty to human review.
I'm sure legora and the other legal AI tech softwares are all over this.
Re: Introducing System One Models and Jev
#437> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle. Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of r…
You need blocking! Fundamentally this is an entity resolution problem. An LLM can score pairwise really well but scoring all the pairs would be insanely computationally difficult. If you can constrain the set of potential matches up front by querying the dataset for things that could be matches it gets a lot more tractable to use an LLM for this. Are there any heuristics you can use to reduce the search space? You me…
The coding agent was pretty good at coming up with heuristics for matching - even more than the dozen I suggested from domain experience. And it used some of them sensibly for blocking, too. I'm sure I could get it to perform a little better and a lot faster with more agent wrangling. I did consider using the heuristics just for blocking, and letting a local LLM do the actual evaluation, but if Jev or Jev-like models work as advertised, maybe we can have the best of both worlds.
Thanks for the link, it is an interesting topic.
Re: Introducing System One Models and Jev
#438this reminds me of laravel boost, which does something similar.
it can generate classes/models/routers etc via tool calls, doesnt write the actual code.
Re: Introducing System One Models and Jev
#439> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle. Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of r…
There's also two other important limitations to using an LLM and just providing it with pairs of records.
It does not know enough about the records in the context of the overall dataset:
- what is the data quality and to what extent do we expect a errors in some fields - how unusual are certain values such as names in the context of the dataset as a whole, e.g. some names would be very common in some countries but rare in others.
I've written in more detail about this here: https://www.robinlinacre.com/fellegi_sunter_accuracy/
Re: Introducing System One Models and Jev
#440Doom demo is beyond impressive, even scary
It's very misleading. If I'm actually playing a game I don't get the coordinates of enemies sent back to me so that I can feed into my mouse to snap my crosshair to. It's looking through walls too, because it's working off structured state in text form. You could re-create this whole demo without using AI. Have an LLM generate the state machine for you and no model is required to run it.
The impressive part is that it is low latency enough to serve high quality answers at game speed through the model instead of a pre generated ad-hoc machine.