Show HN: LLMs can generate valid JSON 100% of the time

Post by simonw » Wed, Aug 16, 2023, 9:57 PM UTC

gsuuon wrote: post #300

simonw wrote: post #194

Earlier quoted context omitted.

I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs. I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.

Even the 7B model is shockingly good! I've been hacking on a project also built on MLC (but the web runtime) and the completions I'm seeing from Llama 2 7B, just running on my laptop's browser, have been really impressive. There's a demo page here: https://ad-llama.vercel.app/

Post by remilouf » Thu, Aug 17, 2023, 9:44 PM UTC

taeric wrote: post #275

Notable that you can't seem to use this trick to have an LLM create JSON that has JSON embedded in it. Which... happens far more often than it probably should. :(

Post by IanCal » Sat, Aug 19, 2023, 9:09 AM UTC

remilouf wrote: post #314

IanCal wrote: post #303

I tried slight modifications from the example pydantic model and it's incredibly slow. Maybe I'm doing something wrong but I've a hefty box and a 3090, an example using gpt-2 doesn't seem like it should be that taxing.

It is currently limited by the time it takes to build the index. There are obvious optimizations we can apply to this, however in a production setting it does not matter much since you only need to build the index once for each (schema, vocabulary) pair.