Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

241–250 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#241

Post by DonHopkins »

BoorishBears wrote: post #189
padolsey wrote: post #101

Earlier quoted context omitted.

XML Elements themselves: their naming, their attributes, comments, indentation. There's more opportunity at every level of the hierarchy to demarkate and establish meaning. Having closing-tags as well, I've found, is a massive boon; LLMs can better understand what "finishing" looks like if its delimited in a semantic way - with a name.

Same works for JSON. Naming JSON keys works for adjusting what the output is nicely, and you can comment in your definitions (by defining them in a JSON Schema, or inserting placeholder text like `"someKeyWithClarifyingDetails": `) I'm actually partial to CSV these days though, it can really cut down on response times just not needing to return all the extra tokens for JSON/XML delimiters

Yikes. This makes me think that JSON's stubborn mistake of not allowing comments is yet another "Billion-Dollar Mistake", since it's way too late to just change the standard to allow comments, update all the JSON content on the internet to use comments, and retrain all the LLMs to understand comments.

Great point about CSVs! But using placeholder keys for JSON comments in untenable, and using schema instead of inline comments is clumsy and indirect. Of course JSON schema are quite useful in certain situations, but LLMs would get a lot more meaning out of casual common JSON if it just allowed comments, and it would also greatly benefit humans.

Between JavaScript's and JSON's mistakes, that's at least THREE BILLION DOLLARS!!! ;)

https://en.wikipedia.org/wiki/Tony_Hoare#Research_and_career

>Speaking at a software conference in 2009, Tony Hoare apologized for inventing the null reference:

>"I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years." -Tony Hoare

https://news.ycombinator.com/item?id=19568378

>"My favorite is always the Billion-Dollar Mistake of having null in the language. And since JavaScript has both null and undefined, it's the Two-Billion-Dollar Mistake." -Anders Hejlsberg

>"It is by far the most problematic part of language design. And it's a single value that -- ha ha ha ha -- that if only that wasn't there, imagine all the problems we wouldn't have, right? If type systems were designed that way. And some type systems are, and some type systems are getting there, but boy, trying to retrofit that on top of a type system that has null in the first place is quite an undertaking." -Anders Hejlsberg

Re: Show HN: LLMs can generate valid JSON 100% of the time

#242

Post by contravariant »

Scene_Cast2 wrote: post #23

One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.

More concretely, sometimes it is not enough to simply constrain the next token, backtracking might end up being better.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#243

Post by kevinlu1248 »

This is amazing! I think for production and rapid development use-cases though we just use XML for information extraction. It's extremely easy to parse with regex and rarely do the models make mistakes since the start and end tokens are uncommon. At least this is just for the OpenAI model which are different from the use cases in this ShowHN.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#244

Post by YeGoblynQueenne »

Hi, remilouf. You say that your background is in "probabilistic, relational and symbolic programming". In that case I suspect you understand that it is no problem to generate text from a regular or context-free grammar, or really any level of grammar. For example, you can do that very easily in Prolog (a relational language) given a grammar in Definite Clause Grammars notation.

As far as I can tell your approach requires a grammar to be given by a user. In that case, what is the advantage of using an LLM to generate text? Why can't you just run your grammar as a generator and generate the text you want? That would save you the considerable trouble and cost of training an LLM in the first place. And why would you need an LLM, a model of natural language, if all you want is to generate structured text, anyway?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#245

Post by BoorishBears »

DonHopkins wrote: post #241
BoorishBears wrote: post #189

Earlier quoted context omitted.

Same works for JSON. Naming JSON keys works for adjusting what the output is nicely, and you can comment in your definitions (by defining them in a JSON Schema, or inserting placeholder text like `"someKeyWithClarifyingDetails": `) I'm actually partial to CSV these days though, it can really cut down on response times just not needing to return all the extra tokens for JSON/XML delimiters

Yikes. This makes me think that JSON's stubborn mistake of not allowing comments is yet another "Billion-Dollar Mistake", since it's way too late to just change the standard to allow comments, update all the JSON content on the internet to use comments, and retrain all the LLMs to understand comments. Great point about CSVs! But using placeholder keys for JSON comments in untenable, and using schema instead of inline…

I'm not saying use placeholder keys: the actual keys themselves serve as guidance.

Naming a key "nameBasedOnLocationIGaveYou" instead of "name", or "oneSentenceSummary" vs "summary", results in a meaningful difference.

You can even use that for formatted single-response chain of thought, like {"listOfStuff":[...], "whatDoTheyHaveInCommon": "", "whichOneIsMostImportant": ""}

Also remember, the LLM doesn't need valid JSON: I just straight up insert comments in the JSON in a non-compliant way for some of my prompts, GPT-4 and Claude are all smart enough to not hallucinate comments back at you. 3.5 might be pushing it if temp is too high (although even the nerfed API logit bias should fix that now that I think about it)

And sometimes to save tokens I describe a JSON object without using JSON: just structure it in neatly formatted markdown and even 3.5 can follow along

Re: Show HN: LLMs can generate valid JSON 100% of the time

#246

Post by msp26 »

panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

>I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten

But you can do both. For my current use case of extracting information from articles, I have a json schema + one/two example articles along with their correct answers. This increases token costs but 3.5 is so cheap that it doesn't matter and for 4 you can use batching to decrease token cost per article.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#247

Post by IanCal »

YeGoblynQueenne wrote: post #244

Hi, remilouf. You say that your background is in "probabilistic, relational and symbolic programming". In that case I suspect you understand that it is no problem to generate text from a regular or context-free grammar, or really any level of grammar. For example, you can do that very easily in Prolog (a relational language) given a grammar in Definite Clause Grammars notation. As far as I can tell your approach requ…

Wouldn't that generate an entirely random but valid output? Here you want a valid output related to the request.

> And why would you need an LLM, a model of natural language, if all you want is to generate structured text, anyway?

So that you can parse unstructured text from a person and return structured data for a machine.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#248

Post by frontier »

OJFord wrote: post #176
frontier wrote: post #171

Earlier quoted context omitted.

Karpathy's series is many many hours long and really does take you from zero to GPT. It's excellent! You sound triggered by the title - that may not even be the official title - but it definitely deserves it. Go look it up.

The title suggests I wouldn't like it, yes. But as a video series it's not 'a textbook or at least written' is i - not really the format I'm looking for personally.

Fair enough if you prefer to slog through an entire textbook. But for anyone else.. I can't recommend this series more highly, it was just amazing, no filler, pure step after step, explained methodically to the end goal.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#249

Post by lefttoreader »

_flux wrote: post #231
lefttoreader wrote: post #165

Earlier quoted context omitted.

Sure! So it’s hopefully clear that the notion of constrained grammar is not novel (see every comment on here of people name-dropping their implementation from two months ago). The novelty here is “instead of checking whether every token is allowed” to create a finite state machine that defines which tokens are allowable at each generation step. This lets them not check every token at every step. The trick of creating…

By "standard regex" stuff I take it you mean the standard regex stuff Python standard library comes with? I mean going from standard regex to NFA to DFA is already more sophisticated than that one, it's _quite_ oldschool and gives you linear time matching: https://en.wikipedia.org/wiki/Thompson%27s_construction https://en.wikipedia.org/wiki/Powerset_construction And what I mean to say by this as they could have easil…

Yep! But that’s sort of my point, and maybe this is just some misplaced academic shit of mine but if you’re going to write a paper then “easily had this idea and never discovered the paper” just doesn’t fly.

Almost all academic work is derivative tweaks of yesterday’s work, yet we still fall over ourselves to cite this stuff.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#250

Post by quickthrower2 »

frontier wrote: post #248
OJFord wrote: post #176

Earlier quoted context omitted.

The title suggests I wouldn't like it, yes. But as a video series it's not 'a textbook or at least written' is i - not really the format I'm looking for personally.

Fair enough if you prefer to slog through an entire textbook. But for anyone else.. I can't recommend this series more highly, it was just amazing, no filler, pure step after step, explained methodically to the end goal.

Lol I had to slog through some traditional material to keep up with Karpathy. A lot is covered in those videos.
Post reply on HN