I am struggling to programmatically get syntactically valid JSON out of LLMs, using both the OpenAI and Vertex apis.
I am using:
"response_format": { "type": "json_object" }
And with Vertex:
"generationConfig": {
"responseMimeType": "application/json"
}
And even:
"response_format": {
"type": "json_schema",
"json_schema": { ...
And with Vertex:
"generationConfig": {
"responseMimeType": "application/json",
"responseSchema": { ...
Neither of them is reliable.
It always gives me json in the format of a markup document with a single json code block:
```json
{}
```
Sure I can strip the code fence, but it's mighty suspicious I asked for json and got markup.
I am getting a huge number of json syntax errors, so it's not even getting to the schemas.
When i did get to the schemas, it was occasionally leaving out fields that I'd declared were required (even if i.e. null or an empty array). So I had to mark them as not required, since the strict schema wasn't guiding it to produce correct output, just catching it when it did.
I admit I'm challenging it by asking it to produce json that contains big strings of markup, which might even contain code blocks with nested json.
If that's a problem, I'll refactor how I send it prompts so it doesn't nest different types.
But that's not easy or efficient, because I need it to return both json and markup in one call, so if I want to use "responseMimeType": "application/json" and "responseSchema", then it can ONLY be json, and the markup NEEDS to be embedded in the json, not the other way around, and there's no way to return both while still getting json and schema validation. I'd hate to have to use tool calls as "out parameters".
But I'm still getting a lot of json parsing problems and schema validation problems that aren't related to nested json formatting.
Are other people regularly seeing markup json code blocks around what's supposed to be pure json, and getting a lot of json parsing and schema validation issues?