We had early access and found it to be pretty useful. Having a second form of verification, where you can ask multiple questions (in the form of Nouls) raised our confidence in the outputs of other models. [0] IMHO This type of model works incredibly well in concert with LLMs, not as a replacement. [0] https://goodstartlabs.com/research/verification-is-the-bottl...
Don't your numbers suggest DeepSeek V4.1 Flash, for $100 more, gets you to slightly better agreement?
Introducing System One Models and Jev
501–510 of 512 posts
Re: Introducing System One Models and Jev
#502I'm curious to hear his take on this approach.
Re: Introducing System One Models and Jev
#503> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle. Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of r…
Re: Introducing System One Models and Jev
#504Couldn't you hook it up to a multiple choice exam?
It's still LLM like, how smart is it? I'm wary of something that the company states they don't want to benchmark it across public benchmarks.
Re: Introducing System One Models and Jev
#505Earlier quoted context omitted.
Not confidential, but not super relevant, as this is something I have learned the hard way over the past year across various projects. A lot of people have become prompt maximalists, asking for complex multi-part solutions or dynamic workflows in a single prompt. You can get this to work sort of reliably with frontier models, but without much confidence or clarity where things might break in practice. My goal is to s…
Example I have worked: Personal delivery app, that tracks packages from various senders using incoming emails. I am using the single prompt approach with GPT5.4, which is free, but it’s not reliable. Using Jev I’d decompose the prompt into a bunch of smaller questions, then I’d combine the answers in software. I’m super excited to try Jev out.
Re: Introducing System One Models and Jev
#506Re: Introducing System One Models and Jev
#507Earlier quoted context omitted.
you don't say - https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD
https://x.com/harshagundal/status/2100044305536889015 tweet by the author
Re: Introducing System One Models and Jev
#508https://x.com/completeskeptic/status/2099925682726002904?s=4... The doom demo is quite cool
I'm not sure the authors realize this is way more than "just a cool demo": if this holds up, it's going to be huge for game QA work. Instrument your game to output properties of entities near the player and the output is the various control inputs - moment to moment gameplay gets solved. Maybe augment with a tick-by-tick controlled stepping mode if particularly twitchy - an LLM can take care of the higher level reaso…