Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Response Healing: Reduce JSON defects by 80%+

openrouter.ai

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Response Healing: Reduce JSON defects by 80%+

#41

Post by wat10000 »

xg15 wrote: post #33
wat10000 wrote: post #32

Earlier quoted context omitted.

Is there anything in the JSON grammer that only allows one valid option? In any case, I also don't understand why it would be costly. The fact that tokens are typically multiple characters would complicate things somewhat, but checking that a given token results in valid partial JSON doesn't seem too hard.

Freeform JSON not so much I think, but if you combine it with a schema and strict whitespace/formatting rules, you could get quite a few. I think there are lots of boilerplate sequences like '":{' or '":[' or '", "', etc - though they might already be compressed into a single token if the tokenizer was trained on enough JSON. There are also situations where the schema would only allow a specific field name as the nex…

Good point, I momentarily forgot about the ability to specify a schema. In that case, you'd have a lot of places where there's only one possible output. Even if you have multiple fields and there are no ordering requirements, typical schemas won't take long to get to a unique field name prefix. If you've output `"us` then `ername"` is likely to be the only valid continuation in many cases.

Re: Response Healing: Reduce JSON defects by 80%+

#42

Post by no_wizard »

Spivak wrote: post #37
blibble wrote: post #30

Earlier quoted context omitted.

it does seem as if the world has gone insane we have brilliant machines that can more or less work perfectly then the scam artists have convinced people that spending a trillion dollar and terawatts to get essentially a biased random number generator to produce unusable garbage is somehow an improvement

These models have turned a bunch of NLP problems that were previously impossible into something trivial. I have personally built extremely reliable systems from the biased random number generator. Our f-score using "classic" NLP went from 20% to 99% using LLMs.

NLP, natural language processing for the unfamiliar. LLMs are tailor made for this work particularly well. They're great tokenizers of structured rules. Its why they're also halfway decent at generating code in some situations.

I think the fall down you see is in logical domains of that rely on relative complexity and contextual awareness in a different way. I've had less luck, for example, having AI systems parse and break down a spreadsheet with complex rules. Thats simply recent memory

Re: Response Healing: Reduce JSON defects by 80%+

#43

Post by Eisenstein »

ksenzee wrote: post #16
idle_zealot wrote: post #14

This really gets at the heart of my instinctive dislike of how LLMs are being deployed. A core feature of computers, and tools in general, is reliability. I like software because you can set something up, run it, and (ideally) know that it will do the same job the same way each subsequent time you run it. I want a button that is clearly labeled, and when pressed, does a specific thing, acting like a limb, an extensio…

> We trained users to treat their devices like unruly animals that they can never quite trust. So now the idea of a machine that embodies a more clever (but still unreliable) animal to wrangle sounds like a clear upgrade. I wish I didn't agree with this, but I think you're exactly right. Even engineers dealing with systems we know are deterministic will joke about making the right sacrifices to the tech gods to get s…

Software is only deterministic if the software it relies on never changes. Forced updates make this impossible, so treating software as deterministic is actually wrong.

Re: Response Healing: Reduce JSON defects by 80%+

#44

Post by fhcuvyxu »

Dylan16807 wrote: post #8
top1aibooster wrote: post #3

> Here's something most developers overlook: if an LLM has a 2% JSON defect rate, and Response Healing drops that to 1%, you haven't just made a 1% improvement. You've cut your defects, bugs, and support tickets in half. If part of my system can't even manage to output JSON reliably, it needs way more "healing" than syntax munging. This comes across as naive.

Plus, that claim isn't even true. A 1% and 2% JSON defect rate are going to annoy a similar amount of people into filing bugs and tickets.

1% fail rate on API requests is horrifyingly embarrassing.

Re: Response Healing: Reduce JSON defects by 80%+

#45

Post by AlexCoventry »

gruez wrote: post #26

>What about XML? The plugin can heal XML output as well - contact us if you’d like access. Isn't this exactly how we got weird html parsing logic in the first place, with "autohealing" logic for mismatched closing tags or quotes?

This is probably a bit different. An LLM outputs a token at a time ("autoregressively") by sampling from a per-position token probability distribution, which depends on all the prior context so far. While the post doesn't describe OpenRouter's approach, most structured LLM output works by putting a mask over that distribution, so that any token which would break the intended structure has probability zero and cannot be sampled. So for instance, in the broken example from the post,

    {"name": "Alice", "age": 30
the standard LLM output would have stopped there because the LLM output an end-of-sequence (EOS) token. But because that would lead to a syntax error in JSON, the EOS token would have probability zero, and it would be forced to either extend the number "30", or add more entries to the object, or end it with "}".

I haven't played much with structured output, but I imagine the biggest risk is that you may force the model to work with contexts outside its training data, leading it to produce garbage, though hopefully syntactically-correct garbage.

I don't understand, though, why the probability of incorrect JSON wouldn't go to 0, under this framework (unless you hit the max sequence length before the JSON ended.) The post implies that JSON errors still happen, so it's possible they're doing something else.

Re: Response Healing: Reduce JSON defects by 80%+

#47

Post by joshstrange »

I’d be (genuinely) interested to hear from people who think this will help. In my mind, if the JSON isn’t valid I wouldn’t trust a “healed” version of it to be correct either. I mean, I guess you just do schema validation on your end and so maybe fixing a missing comma/brace/etc is actually really helpful. I’ve not done JSON generation at scale to know.

Re: Response Healing: Reduce JSON defects by 80%+

#48

Post by red2awn »

osaariki wrote: post #39
red2awn wrote: post #23

Very confused. When you enable structured output the response should adhere to the JSON schema EXACTLY, not best effort, by constraining the output via guided decoding. This is even documented in OpenRouter's structured output doc > The model will respond with a JSON object that strictly follows your schema Gemini is listed as a model supporting structured output, and yet its fail rate is 0.39% (Gemini 2.0 Flash)!! I…

You're exactly right. The llguidance library [1,2] seems to have emerged as the go-to solution for this by virtue of being >10X faster than its competition. It's work from some past colleagues of mine at Microsoft Research based on theory of (regex) derivatives, which we perviously used to ship a novel kind of regex engine for .NET. It's cool work and AFAIK should ensure full adherence to a JSON grammar. llguidance i…

Cool stuff! I don't get how all the open source inference framework have this down but the big labs doesn't...

Gemini [0] is falsely advertising this:

> This capability guarantees predictable and parsable results, ensures format and type-safety, enables the programmatic detection of refusals, and simplifies prompting.

[0]: https://ai.google.dev/gemini-api/docs/structured-output?exam...

Re: Response Healing: Reduce JSON defects by 80%+

#49

Post by red2awn »

xg15 wrote: post #31
numlocked wrote: post #11

Earlier quoted context omitted.

That is a way of doing that, but it's quite expensive computationally. There are some companies that can make it feasible [0], but it's often not a perfect process and different inference providers implement it different ways. [0] https://dottxt.ai/

Out of curiosity, why is it so expensive? Shouldn't constraining the possible result tokens make the inference less expensive? (because you have to calculate less logits and could occasionally even skip tokens entirely if there is only one valid option)

Tokens are sampled from logits using the constraints after a normal forward pass. The forward pass is the expensive part of LLM inference which isn't affected by structured output.

Re: Response Healing: Reduce JSON defects by 80%+

#50

Post by xg15 »

red2awn wrote: post #49
xg15 wrote: post #31

Earlier quoted context omitted.

Out of curiosity, why is it so expensive? Shouldn't constraining the possible result tokens make the inference less expensive? (because you have to calculate less logits and could occasionally even skip tokens entirely if there is only one valid option)

Tokens are sampled from logits using the constraints after a normal forward pass. The forward pass is the expensive part of LLM inference which isn't affected by structured output.

Yes, but if the constraints only permit a single valid token anyway for some positions, you could skip the forward pass entirely for those positions and just return that token.

The other idea was a bit more theoretical: If you know only a handful tokens are valid, then calculating the logits of the other tokens in the forward pass is wasteful as they won't affect the sampling process. However, it's probably not worthe the cost to optimize that as it only affects the last layer and might be mostly amortized by SIMD parallel processing anyway.

Post reply on HN