31–40 of 89 posts
1. Langchain not being used in production?
> How out of touch is that remark? Hard pressed to find agentic framework implementation outside of Langchain/Llamaindex.
2. Outlines is not expected to work with OpenAI API because it wasn't created to do that.
Those LLM not only suck megawatts of energy and TFLOPS of compute, but they also consume heaps of brain power - all that for what, in the end? What betterment?
This is an article written by BAML that shows BAML as the best. Also, BAML seems to be a commercial product with no clear pricing. > Our paid capabilities only start if you use Boundary Studio, which focuses on Monitoring, Collecting Feedback, and Improving your AI pipelines. Contact us for pricing details at contact_boundaryml.com
> Also, BAML seems to be a commercial product with no clear pricing. You've only presented half the story. They're also Open Source (Apache 2.0), with code on github. As you mention, some features are gated, but they seem to have a fairly solid OSS offering.
We just use openai function calls (tools) and then use Pydantic to verify the JSON. When validation fails we try the prompt again.
[Other BAML creator here!] one time we told a customer to do this to fix small json mistakes but turns out their customers don't tolerate a +20-30s increase in latency for regenerating a long json structure. We instead had to write a parser to catch small mistakes like missing commas, quotes etc, and parse content even if there's things like reasoning in the response, like here: https://www.promptfiddle.com/Chain-of-…
Can't we ask LLM/GenAI to summarize it and return structured output? /sarcasm (half?)
{ author: "Sam Lijin"
key_points: [ "Structured output from LLMs, like JSON, is a common challenge."
"Existing solutions like response_format: 'json' and function calling often disappoint."
"The article compares multiple frameworks designed to handle structured output."
"Handling and preventing malformed JSON is a critical concern."
"Two main techniques for this: parsing malformed JSON or constraining LLM token generation."
"Framework comparison includes details on language support, JSON handling, prompt building, control, model providers, API flavors, type definitions, and test frameworks."
"BAML is noted for its robust handling of malformed JSON using a new Rust-based parser."
"Instructor supports multiple LLM providers but has limitations on prompt control."
"Guidance, Outlines, and others apply LLM token constraints but have limitations with models like OpenAI's."take_way: "Consider using frameworks that efficiently handle malformed JSON and offer prompt control to get the desired structured output from LLMs."
}
We just use openai function calls (tools) and then use Pydantic to verify the JSON. When validation fails we try the prompt again.
Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…
AI noob question: Why do OpenAI/Anthropic/... not support constraining token generation? I'd imagine producing valid structured output would be at the top of their feature request lists.
LLMs today are really good at producing output that satisfies the very vague metric of “this looks good to a human” but aren’t nearly as good at producing output that satisfies a complex set of syntax and schema constraints. The state space of the former is much larger than the latter, so there’s a lot more opportunity for an LLM to be successful by targeting the state space of “looks good to a human”. Plus, there’s still a lot of room for advancement in multimodality and data quality improvements.
Search problems, in general, deal with this too: it’s easy to provide a good search experience when there are a lot of high-quality candidates, and much harder when there are fewer, because all you have to do is return just a few of the best candidates. (This is partly why Google Drive Search has always sucked compared to Web Search- it’s really hard to guess exactly which document in a 10k-file-Drive a user is looking for, as opposed to finding something on Wikipedia/NYTimes/Instagram that the user might be looking for!)