41–50 of 89 posts
>> “you've tried response_format: "json" and function calling and been disappointed by the results”
Can anyone share any examples of disappointments or issues with these techniques? Overall I’ve been pretty happy with JSON mode via OpenAI API so I’m curious to hear about any drawbacks with it.
Seems like a marketing post to me. Langchain and Llama-Index are indeed used in production. Where does it state that the other libraries are used in production?
Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…
Isn't reprompting a decent technique? Considering most modern languages are LL(k), that is you need at most k tokens to parse the output (tbf these are programming language tokens not LLM tokens), with k=1 being the most common choice, would it not be reasonable to expect to only have to regenerate only a handful of tokens at most?
If you’re driving user-facing interactions with LLMs, though, and you’re already dealing with >1min latency on the first call (as many of our current users are!), waiting for another LLM call to come back is a really frustrating thing to block your UX on.
The article mentions, >> “you've tried response_format: "json" and function calling and been disappointed by the results” Can anyone share any examples of disappointments or issues with these techniques? Overall I’ve been pretty happy with JSON mode via OpenAI API so I’m curious to hear about any drawbacks with it.
- Return a single object instead of an array of objects
- Return an array of a single object instead of just the object
On the other hand I personally haven't seen it give malformed JSON; the JSON is well-formed but not compliant with the schema we specified.
The article mentions, >> “you've tried response_format: "json" and function calling and been disappointed by the results” Can anyone share any examples of disappointments or issues with these techniques? Overall I’ve been pretty happy with JSON mode via OpenAI API so I’m curious to hear about any drawbacks with it.
forcing those parts to be json, can be hard and unnecessarily constrain the model. e.g. https://www.promptfiddle.com/Chain-of-Thought-KcSBh
try pressing run tests and you'll see what i mean! this method or doing chain of thought works a bit better
Earlier quoted context omitted.
[Other BAML creator here!] one time we told a customer to do this to fix small json mistakes but turns out their customers don't tolerate a +20-30s increase in latency for regenerating a long json structure. We instead had to write a parser to catch small mistakes like missing commas, quotes etc, and parse content even if there's things like reasoning in the response, like here: https://www.promptfiddle.com/Chain-of-…
I'm not sure I understand, in the docs for the python client it says that BAML types get converted to Pydantic models, doesn't that step include the extra latency you mentioned?
- retry the request, which may take 30+ secs (if your LLM outputs are really long and you're using something like gpt4)
- fix the parsing issue
In our library we do the latter. The conversion from BAML types to Pydantic ones is a compile-time step unrelated to the problem above. That doesn't happen at runtime.
The article mentions, >> “you've tried response_format: "json" and function calling and been disappointed by the results” Can anyone share any examples of disappointments or issues with these techniques? Overall I’ve been pretty happy with JSON mode via OpenAI API so I’m curious to hear about any drawbacks with it.
We specify an output schema (TypeScript syntax) in our system prompt, and OpenAI gets it right most of the time. With some regularity it will give invalid (per the schema) output like - Return a single object instead of an array of objects - Return an array of a single object instead of just the object On the other hand I personally haven't seen it give malformed JSON; the JSON is well-formed but not compliant with t…
fyi, we actually fix those specific errors in our parser :)
The baml config files look a lot like code. For example in baml: class Resume { name string education Education[] @description("Extract in the same order listed") skills string[] @description("Only include programming languages") } Could be expressed in Python like this: class Resume: name: str education: List[Education] # Extract in the same order listed skills: List[str] # Only include programming languages Two ben…
https://docs.pydantic.dev/latest/concepts/models/#dynamic-mo...
This is an article written by BAML that shows BAML as the best. Also, BAML seems to be a commercial product with no clear pricing. > Our paid capabilities only start if you use Boundary Studio, which focuses on Monitoring, Collecting Feedback, and Improving your AI pipelines. Contact us for pricing details at contact_boundaryml.com
That being said, if you have suggestions for how we can make this table more objective, we’re all ears!
Earlier quoted context omitted.
I'm not sure I understand, in the docs for the python client it says that BAML types get converted to Pydantic models, doesn't that step include the extra latency you mentioned?
My bad, I think I didnt explain correctly. Basically you have two options when a "," is missing (amongst other issues) in an LLM output which causes a parsing issue: - retry the request, which may take 30+ secs (if your LLM outputs are really long and you're using something like gpt4) - fix the parsing issue In our library we do the latter. The conversion from BAML types to Pydantic ones is a compile-time step unrela…