Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Response Healing: Reduce JSON defects by 80%+

openrouter.ai

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Response Healing: Reduce JSON defects by 80%+

#31

Post by xg15 »

numlocked wrote: post #11
wat10000 wrote: post #10

I thought structured output was done by only allowing tokens that would produce valid output. For their example of a missing closing bracket, the end token wouldn't be allowed, and it would only accept tokens that contain a digit, comma, or closing bracket. I guess that must not be the case, though. Doing that seems like a better way to address this.

That is a way of doing that, but it's quite expensive computationally. There are some companies that can make it feasible [0], but it's often not a perfect process and different inference providers implement it different ways. [0] https://dottxt.ai/

Out of curiosity, why is it so expensive? Shouldn't constraining the possible result tokens make the inference less expensive? (because you have to calculate less logits and could occasionally even skip tokens entirely if there is only one valid option)

Re: Response Healing: Reduce JSON defects by 80%+

#32

Post by wat10000 »

xg15 wrote: post #31
numlocked wrote: post #11

Earlier quoted context omitted.

That is a way of doing that, but it's quite expensive computationally. There are some companies that can make it feasible [0], but it's often not a perfect process and different inference providers implement it different ways. [0] https://dottxt.ai/

Out of curiosity, why is it so expensive? Shouldn't constraining the possible result tokens make the inference less expensive? (because you have to calculate less logits and could occasionally even skip tokens entirely if there is only one valid option)

Is there anything in the JSON grammer that only allows one valid option? In any case, I also don't understand why it would be costly. The fact that tokens are typically multiple characters would complicate things somewhat, but checking that a given token results in valid partial JSON doesn't seem too hard.

Re: Response Healing: Reduce JSON defects by 80%+

#33

Post by xg15 »

wat10000 wrote: post #32
xg15 wrote: post #31

Earlier quoted context omitted.

Out of curiosity, why is it so expensive? Shouldn't constraining the possible result tokens make the inference less expensive? (because you have to calculate less logits and could occasionally even skip tokens entirely if there is only one valid option)

Is there anything in the JSON grammer that only allows one valid option? In any case, I also don't understand why it would be costly. The fact that tokens are typically multiple characters would complicate things somewhat, but checking that a given token results in valid partial JSON doesn't seem too hard.

Freeform JSON not so much I think, but if you combine it with a schema and strict whitespace/formatting rules, you could get quite a few.

I think there are lots of boilerplate sequences like '":{' or '":[' or '", "', etc - though they might already be compressed into a single token if the tokenizer was trained on enough JSON.

There are also situations where the schema would only allow a specific field name as the next token, e.g. if it was the only remaining valid and required field, or if fields have to be output in a specific order.

Re: Response Healing: Reduce JSON defects by 80%+

#34

Post by Spivak »

top1aibooster wrote: post #3

> Here's something most developers overlook: if an LLM has a 2% JSON defect rate, and Response Healing drops that to 1%, you haven't just made a 1% improvement. You've cut your defects, bugs, and support tickets in half. If part of my system can't even manage to output JSON reliably, it needs way more "healing" than syntax munging. This comes across as naive.

Model itself can't output JSON reliably. It's on you building a system around the model to make sure it either returns correct output or errors which is trivial to do.

Re: Response Healing: Reduce JSON defects by 80%+

#35

Post by Eisenstein »

oats wrote: post #28

Is this a joke? Am I going crazy? I don't like this future we're going towards where we have to trick our software (which we can no longer understand the workings of) into doing what we tell it to by asking it nicely, or by putting another black box on the end to "fix" the output. This is the opposite of engineering. This is negotiation with a genie trapped in silicon.

It is easier to realize that software development was never engineering. Physical engineering is reliant on physics, while software is reliant on other software. Physics are static and as regarding practical engineering is known and can be applied rigorously and taught in courses. Software is constantly changing, contain tons of edge cases, and as we can see by recent developments, can change in unpredictable ways and lead to entirely new paradigms.

So, the software that you learned on is changing. You aren't going crazy, but the ground is indeed shifting. The problem is that you assumed it couldn't shift because you were applying the wrong constraints.

Re: Response Healing: Reduce JSON defects by 80%+

#36

Post by curtisf »

Sabinus wrote: post #24
nubg wrote: post #19

Earlier quoted context omitted.

Meaning parts were LLM written? With no disclosure?

"With no disclosure?" Why do you have an expectation that a company will disclose to you when they use AI for their copywriting? Do you want them to disclose the software they used to draft and publish? If a manager reviewed the blog post before it went live?

Using words written by other people without disclosure has always been frowned upon. It's called plagiarism.

Plagiarism is bad for a lot of reasons, all of which also apply to the undisclosed use of generative AI.

Re: Response Healing: Reduce JSON defects by 80%+

#37

Post by Spivak »

blibble wrote: post #30
oats wrote: post #28

Is this a joke? Am I going crazy? I don't like this future we're going towards where we have to trick our software (which we can no longer understand the workings of) into doing what we tell it to by asking it nicely, or by putting another black box on the end to "fix" the output. This is the opposite of engineering. This is negotiation with a genie trapped in silicon.

it does seem as if the world has gone insane we have brilliant machines that can more or less work perfectly then the scam artists have convinced people that spending a trillion dollar and terawatts to get essentially a biased random number generator to produce unusable garbage is somehow an improvement

These models have turned a bunch of NLP problems that were previously impossible into something trivial. I have personally built extremely reliable systems from the biased random number generator. Our f-score using "classic" NLP went from 20% to 99% using LLMs.

Re: Response Healing: Reduce JSON defects by 80%+

#38

Post by spwack »

top1aibooster wrote: post #3

> Here's something most developers overlook: if an LLM has a 2% JSON defect rate, and Response Healing drops that to 1%, you haven't just made a 1% improvement. You've cut your defects, bugs, and support tickets in half. If part of my system can't even manage to output JSON reliably, it needs way more "healing" than syntax munging. This comes across as naive.

[dead]

Re: Response Healing: Reduce JSON defects by 80%+

#39

Post by osaariki »

red2awn wrote: post #23

Very confused. When you enable structured output the response should adhere to the JSON schema EXACTLY, not best effort, by constraining the output via guided decoding. This is even documented in OpenRouter's structured output doc > The model will respond with a JSON object that strictly follows your schema Gemini is listed as a model supporting structured output, and yet its fail rate is 0.39% (Gemini 2.0 Flash)!! I…

You're exactly right. The llguidance library [1,2] seems to have emerged as the go-to solution for this by virtue of being >10X faster than its competition. It's work from some past colleagues of mine at Microsoft Research based on theory of (regex) derivatives, which we perviously used to ship a novel kind of regex engine for .NET. It's cool work and AFAIK should ensure full adherence to a JSON grammar.

llguidance is used in vLLM, SGLang, internally at OpenAI and elsewhere. At the same time, I also see a non-trivial JSON error rate from Gemini models in large scale synthetic generations, so perhaps Google hasn't seen the "llight" yet and are using something less principled.

1: https://guidance-ai.github.io/llguidance/llg-go-brrr 2: https://github.com/guidance-ai/llguidance

Post reply on HN