Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

141–150 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#141

Post by itissid »

I have noob thought on the potential of these in Formal path planning. Specifically given a set of functions that basically map {State -> Actions} given preconditions, transition functions (heavily paraphrasing STRIPS[1]) can a correct and optionally "realistic" plan be generated[2]? I am quite interested in this. It seems clear that the issue is that there is no "guidance" like DFA on what is the correct next symbol for a Plan, but perhaps the AI can generate some kind of a probability or order on what is the best step and one can go from there...

Are you guys thinking about this direction?

[1] https://en.wikipedia.org/wiki/Stanford_Research_Institute_Pr...

[2] Formal Planning decision problem(plan exists) given STRIPS spec is at least NP-Complete[1]. There are several mathematical, logical and statistical "tricks"(e.g. [3]) that are used to bring down the complexity and try find a plan using heuristics(thinking MDPs, POMDPs here). This is not new, everyone in LLM research knows this.

[3] "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning": https://www.sciencedirect.com/science/article/pii/S000437029...

Re: Show HN: LLMs can generate valid JSON 100% of the time

#142

Post by darkteflon »

anotherpaulg wrote: post #131

For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code editing benchmarks of GPT-3.5 and GPT-4: https://aider.chat/docs/benchmarks.html I’m curious if you have measured whether the “constrained generation” that you’re doing suffers from similar downsides?

We’ve seen this too. We run them as two separate stages - “reason”, log the intermediate output, then parse.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#143

Post by quickthrower2 »

dwattttt wrote: post #139
quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

If you're choosing the next token based on a list of valid next tokens, a uniform random distribution can always generate valid JSON too!

Yep. So can this:

    fun generate_valid_json(seed):
        return "{}"

Re: Show HN: LLMs can generate valid JSON 100% of the time

#144

Post by bestcoder69 »

chipsrafferty wrote: post #81
coder543 wrote: post #44

Earlier quoted context omitted.

This analogy falls apart because the spellchecker is separate from the author, and doesn’t know what the author intended. Here, the LLM is still dictating the token probabilities, so the content will be as correct as the LLM can make it, given the constraints. AIUI, the sampler is just choosing tokens on a combination of probability and syntactic correctness, instead of strictly on probability. If the LLM is forced t…

Why isn't it possible to design LLMs that say "I don't know"?

text-davinci-002 used to make me so mad with how often it’d do that

Re: Show HN: LLMs can generate valid JSON 100% of the time

#145

Post by coder543 »

quickthrower2 wrote: post #138
mattigames wrote: post #135

Earlier quoted context omitted.

It's not like humans are particularly good at distinguishing truth from lies.

The word "lie" is probably too anthropic here. I should have just said "made up". There is no intent to lie. And the model isn't try to self-fact-check anyway. (Maybe some do). But if they do they are probably bad at it at the moment, at least from my experience of GPT3.5 (not used 4 much).

> at least from my experience of GPT3.5 (not used 4 much).

And 4 is tremendously better than 3.5, in my own experience. Not perfect, but actually useful.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#146

Post by dceddia »

huevosabio wrote: post #108
nabakin wrote: post #89

Earlier quoted context omitted.

Don't rely too much on automated benchmarks for LLMs. They are often gamed, made to overfit, and result in worse performance in the general case. Human evaluation is the gold standard and the Llama 2 paper gave significant evidence that Llama 2 70b chat is on-par, if not, better than ChatGPT for that metric so I tend to stick to it unless there is good reason not to.

The problem with Llama 2 chat versions is that they have been RLHF-ed to death. You can't ask questions without getting a sermon of how your question may be inappropriate for this or that reason. I think it's worse on the smaller models, but still present in the 70B one.

Apologies if you’d already seen this and were only trying to make a point, but you might like this article from a week or 2 ago that talks about how to run Llama 2 “uncensored” locally, and it seems to do a decent job of mitigating the sermons!

Article: https://ollama.ai/blog/run-llama2-uncensored-locally

Discussion: https://news.ycombinator.com/item?id=36973584

Re: Show HN: LLMs can generate valid JSON 100% of the time

#148

Post by Scaevolus »

Are there temperature or sampling parameters for generate.regex? I'm poking around trying to generate password mnemonics (https://rmmh.github.io/abbrase/), and it really doesn't like actually giving me proper words:

    >> model = models.transformers("gpt2-medium")
    >> generate.regex(model, r"Rea[a-z']{,10} lik[a-z']{,10} acr[a-z']{,10} ene[a-z']{,10} sta[a-z']{,10}\.", max_tokens=30)("A memorable phrase is:")
    'Rearmingandme like acrowetteanda eneatubootank stackfishkies.'

Re: Show HN: LLMs can generate valid JSON 100% of the time

#149

Post by verdverm »

lefttoreader wrote: post #110

The “trick” seems to blatantly rip off FlashText without citing it? https://arxiv.org/pdf/1711.00046.pdf I’m a fan of the approach. I normally wouldn’t care if this was just another LLM library taking inspiration, but if you’re going to go out of your way to put a paper on the ArXiv, feels like doing a literature review is a good step?

Care to explain how a string replacement algorithm relates to nudging the logits of a ML model?

I don't see the "rip off", the paper you cite requires a complete document to work on while this work is for guiding the generation of tokens

Post reply on HN