Viewing profile — FlyingLawnmower

Very nicely written guide! If the authors or readers are interested in some of the more technical details of how we optimized guidance & llguidance, we wrote up a little paper abou…

Good point re: documentation... We have support for Huggingface Transformers, llama.cpp, vLLM, SGLang, and TensorRT-LLM, along with some smaller providers (e.g. mistral.rs). Using …

guidance can handle many context-free grammars. We use an Earley parser under the hood ( https://en.wikipedia.org/wiki/Earley_parser ) which gives us significant flexibility boosts…

We did quite a thorough benchmarking of various structured decoding providers in one of our papers: https://arxiv.org/abs/2501.10868v3 , measuring structured outputs providers on p…

Thanks :) Great question re: adoption...it's definitely dominated by JSON. Most API providers have standardized on JSON outputs, so application teams have started building shims th…

If you can screen tokens against your grammar fast enough, you can build a bitmask over the entire token vocabulary and apply it right before sampling. As vocabulary sizes grow, th…

If your masking is fast enough, you can make it easily work with spec dec too :). We manage to keep this on CPU. Some details here: https://github.com/guidance-ai/llguidance/blob/m…

OpenAI has started to (at least for tool calls): https://platform.openai.com/docs/guides/function-calling#con...

We do enable forcing these sequences of tokens in guidance, and find that it significantly speeds up structured generation. There are tricky alignment issues to make sure you pick …

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy…

Sidenote, but the scholarship on distillation always makes me a bit sad. The Original work, cited in the abstract of the Hinton, Vinyals, and Dean paper that is cited everywhere, w…

Yes. They are about entirely different patient reports.

Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes an…

https://chatgpt.com/share/6760a122-0ec4-8008-8b72-3e950f0288... My first try with o1. Seems right to me…what does this teach us about LLMs :)?

Interesting paper, but their reason for dismissing constrained decoding methods seems to be that they want to academically study the in-context setting. For practitioners, using a …

Last I checked, physician pay represents a fairly small (single digit) fraction of healthcare expenditure. If I recall right, administrative overhead and insurance is significantly…

Late to the thread here, but the paper announcing Med-PaLM ( https://arxiv.org/abs/2212.13138 ) does not report many benchmark results on Med-PaLM and is instead mostly about Flan-…

"We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not spe…

Do you have some sources where I can read up more on this? I have always believed that the cost of front line worker salaries (Doctors/Nurses) was a relatively small % of the total…

Have you thought about using central/global differential privacy (which tends to have much less noise) on the "high level aggregates" or "aggregated datasets" that persist after th…