Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

301–310 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#301

Post by selcuka »

tomduncalf wrote: post #272
ipaddr wrote: post #207

Earlier quoted context omitted.

You can change the randomness value to 0 and get the same output each time for the same text

In my experience (with GPT-4 at least), a temperature of 0 does not result in deterministic output. It's more consistent but outputs do still vary for the same input. I feel like temperature is a bit more like "how creative should the model be?"

One theory is it is caused by its Sparse MoE (Mixture of Experts) architecture [1]:

> The GPT-4 API is hosted with a backend that does batched inference. Although some of the randomness may be explained by other factors, the vast majority of non-determinism in the API is explainable by its Sparse MoE architecture failing to enforce per-sequence determinism.

[1] https://152334h.github.io/blog/non-determinism-in-gpt-4/

Re: Show HN: LLMs can generate valid JSON 100% of the time

#302

Post by IanCal »

YeGoblynQueenne wrote: post #297
IanCal wrote: post #293

Earlier quoted context omitted.

> As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). You wouldn't need to, that's the point here. You let the LLM work on generating semantically valid responses a…

>> Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). That'd be nice, but it's not how this tool works. If you look at the repo, there's an example of following a json schema or pydantic model. It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: class Weapon(str, Enum): sword = "sword" axe = "axe" mace = "mace" spear = "sp…

> It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand:

No, only if you want to explicitly limit it to a set of options. You can have freeform fields, just like the jsonschema I provided. If you look at the example there's a character name which has a constrained length but is not limited to a set of options:

    class Character(BaseModel):
        name: constr(max_length=10)
        age: int
        armor: Armor
        weapon: Weapon
        strength: int
The name there can be anything you want. This tool is, unfortunately, outrageously slow so I put the json schema above with a few fixes into jsonformer and downloaded a small model and used it to convert the GPT4 description into valid json:

    {
    "backstory":"Born in the tranquil meadows of Veggie",
    "weapons":[
        {
            "name":"Leek Lance",
            "description":"A long, green and white lance made from a leek",
            "weapon_type":"distance",
            "range":100.0,
            "damage":75.0
        },
        {
            "name":"Carrot Dagger",
            "description":"A short, pointed dagger. It's sharp",
            "weapon_type":"close",
            "range":10.0,
            "damage":50.0
        }
    ],
    "name":"Sir Turnip Thistlebrook"
    }
> Not arbitrary.

Well exactly. If you want to support arbitrary requests while constraining the output, tools like this are an easy approach and I'm not sure what else comes close. An interactive character design flow would have something like the above as the defined output and you could just keep asking for alterations as a human would ("make it more whimsical" or "not a king, something lower class") and have useful structured output

> See my casting-to-type analogy. The point I'm trying really hard to get across is that generating free-form text is all nice and cool, but if you want to give it structure, you need to have the entire structure defined before-hand, otherwise the text that can't be made to conform to it simply won't.

The structure, sure. But the content can be extremely varied.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#304

Post by nl »

victor106 wrote: post #274
nl wrote: post #223

Earlier quoted context omitted.

Not simonw, but I've been using Llama2-13B for search re-ranking very successfully.

search re-ranking?

Do a search, then re-order the results based on a criteria. Easy when the criteria is easy to code, less so when it isn't. But turns out LLMs are pretty good at interpreting the re-ranking instructions.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#305

Post by YeGoblynQueenne »

IanCal wrote: post #302
YeGoblynQueenne wrote: post #297

Earlier quoted context omitted.

>> Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). That'd be nice, but it's not how this tool works. If you look at the repo, there's an example of following a json schema or pydantic model. It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: class Weapon(str, Enum): sword = "sword" axe = "axe" mace = "mace" spear = "sp…

> It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: No, only if you want to explicitly limit it to a set of options. You can have freeform fields, just like the jsonschema I provided. If you look at the example there's a character name which has a constrained length but is not limited to a set of options: class Character(BaseModel): name: constr(max_length=10) age: int…

Thanks for the demonstration. Well, maybe I did understimate the tool after all, although I'd prefer to see the entire session (prompt, grammar, and all the interactions) to be fully convinced.

I suspect though that the reason the tool was "outrageously slow" in your experiment is that you gave a very general grammar. Constraining it more (by giving exact descriptions of weapons) would perhaps make it work faster.

Also, it's obvious that while you'll get valid json like that, you have no guarantee that the contents will always match your request. This time you got a carrot dagger (again- I'd like to see the prompt that led to that, please), next time you might not.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#306

Post by OJFord »

quickthrower2 wrote: post #252
OJFord wrote: post #238

Earlier quoted context omitted.

I just don't find it an effective way of learning personally. I didn't expect this to be so controversial - different people learn differently.

I recommend this for groundwork to get you near LLM, and cover the journey deeply. I used some of this as a helper course for Karpathy. I learned things here he didn’t cover and vice versa. https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/ I haven’t done tonnes of courses so there might be better. But this is good as a free one.

Thanks!

Re: Show HN: LLMs can generate valid JSON 100% of the time

#307

Post by IanCal »

YeGoblynQueenne wrote: post #305
IanCal wrote: post #302

Earlier quoted context omitted.

> It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: No, only if you want to explicitly limit it to a set of options. You can have freeform fields, just like the jsonschema I provided. If you look at the example there's a character name which has a constrained length but is not limited to a set of options: class Character(BaseModel): name: constr(max_length=10) age: int…

Thanks for the demonstration. Well, maybe I did understimate the tool after all, although I'd prefer to see the entire session (prompt, grammar, and all the interactions) to be fully convinced. I suspect though that the reason the tool was "outrageously slow" in your experiment is that you gave a very general grammar. Constraining it more (by giving exact descriptions of weapons) would perhaps make it work faster. Al…

Happy to help, my current focus is on llms and how to understand them (pros, cons, how to use them safely and where they can fit into your workflow) so opportunities to talk through these things are useful for me.

> I suspect though that the reason the tool was "outrageously slow" in your experiment is that you gave a very general grammar

Actually even smallish ones caused problems but jsonformer (a similar tool) worked fine. Not sure what the issue is with this one, I couldn't get it to complete. Not sure if I've got the hacked together code I used to get the json, I was using very small models which didn't help but my internet is slow and I couldn't load anything decent in the time so some of the testing was "here's an llms jsonish output, fix it to this exact schema". Smaller models needed more hand holding. Gpt2 had no idea how to deal with it.

For jsonformer the grammar was near identical to what I posted before, I fixed a couple of typos I think.

Personally the flow of:

Reason about the problem

Write in english

Convert to JSON

- use a tool like this to fix broken JSON

Is a workflow I think is very applicable (you can use different models for any step too).

> again- I'd like to see the prompt that led to that, please

Sure, that was from gpt4, which actually was either fine or decent if given the jsonschema.

Here's the original prompt and the full response that had a full backstory:

> fun but not over the top character from the middle ages, with relevant weapons and a backstory. Game theme is a world populated by anthropomorphic vegetables

https://chat.openai.com/share/4037c8b3-d1bf-4e66-b98d-b518aa...

It's a shame you can't use some of these tools with gpt4, it's in a class of its own.

> Also, it's obvious that while you'll get valid json like that, you have no guarantee that the contents will always match your request

Yeah absolutely. You need to be doing something simple enough for the llm in use to reliably generate sensible output, tools like this then let you integrate that into other systems. How best to use llms really comes into how to pick a good one for the use case and how critical errors are - proposing d&d characters is a very low risk option (human oversight, no automatic application, errors are mostly just annoying, fixing is easy).

Re: Show HN: LLMs can generate valid JSON 100% of the time

#308

Post by RyEgswuCsn »

MereInterest wrote: post #163
RyEgswuCsn wrote: post #154

Earlier quoted context omitted.

How does the LLM know what valid JSON tokens are? What if the training data contains malformed JSON? There ought to be a non-zero chance of the LLM producing invalid JSON, no?

LLMs work by outputting a value for each token, then using those values to generate a probability distribution. Usually, this will be through a function like softmax [0], but there's nothing preventing you from doing some post-processing first. That post processing could be aware of the tokens that would be valid as the next token in a JSON format, and set the probabilities of all other tokens to zero. That way, even…

But that is not the LLM learning to produce valid JSON —— as some other commenters mentioned, you can get valid json without the LLM.

Sure it could be useful, but not really impressive.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#309

Post by baobabKoodaa »

Kiro wrote: post #232
baobabKoodaa wrote: post #106

Earlier quoted context omitted.

No. You need to hook into the LLM at a lower level. One API call typically triggers a generation of a sequence of tokens and this library has to poke into things between each generated token.

Can't I use the max_tokens (set to 1) and logit_bias parameters? Not saying I want to do this. I just want to understand how this works.

Not sure exactly what is logit_bias, but after Googling for 5 seconds it seems to be an OpenAI parameter that's not available in HuggingFace transformers?

Anyway, if your idea is to make one API call per token, the biggest problem with that approach is that it would be really slow to do that.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#310

Post by caesil »

selcuka wrote: post #172
caesil wrote: post #104

Earlier quoted context omitted.

With ChatGPT function calling I get valid JSON 100% of the time from GPT-4 unless I have made some error in prompting. The chief error is not providing escape hatches. LLMs look for a right answer. If you are feeding it some texts and asking it to return structured data about the texts, but then one of the texts is blank, it will be difficult to determine a right answer, so you get hallucinations. The solution is an…

The premise of function calling is great, but in my experience (at least on GPT-3.5, haven't tried it with GPT-4 yet) it seems to generate wildly different, and less useful results, for the same prompt.

GPT-3.5 is pretty much useless for reliable NLP work unless you give it a VERY proscribed task.

That's really the major breakthrough of GPT-4, in my mind, and the reason we are absolutely going to see an explosion of AI-boosted productivity over the next few years, even if foundation LLM advancements stopped cold right now. A vast ocean of mundane white collar work is waiting to be automated.

Post reply on HN