Show HN: VLLM with JSON Guided Generation

#1

Post by btwillard » Fri, Dec 22, 2023, 4:12 PM UTC

Our project, Outlines, now offers guided/constrained generation (e.g. according to a JSON schema) via the VLLM library.

My colleague, Rémi, created some patches that allow one to pass vLLM a JSON schema along with the prompt, which dramatically simplifies deployment of JSON-guided generation. He also added a new `serve` interface that puts it all together and makes serving such models a 2-3 line process.

Check it out and tell us what you think!

Show HN: VLLM with JSON Guided Generation
outlines-dev.github.io

Re: Show HN: VLLM with JSON Guided Generation

#2

Post by jilijeanlouis » Fri, Dec 22, 2023, 4:17 PM UTC

this is really cool probably the most wanted feature in LLM open source. It would be nice to have regex support too?

do you plan to support regex?

Re: Show HN: VLLM with JSON Guided Generation

#3

Post by remilouf » Fri, Dec 22, 2023, 4:20 PM UTC

jilijeanlouis wrote: post #2

this is really cool probably the most wanted feature in LLM open source. It would be nice to have regex support too? do you plan to support regex?

We already support regex-guided generation in the library, and could easily make an API to serve this as well if that's a feature people want!

Re: Show HN: VLLM with JSON Guided Generation

#4

Post by jilijeanlouis » Fri, Dec 22, 2023, 4:26 PM UTC

remilouf wrote: post #3

jilijeanlouis wrote: post #2

this is really cool probably the most wanted feature in LLM open source. It would be nice to have regex support too? do you plan to support regex?

We already support regex-guided generation in the library, and could easily make an API to serve this as well if that's a feature people want!

I need it yes. that would be amazing tbh.