LLMs are like a box of chocolates...
Using “underdrawings” for accurate text and numbers
41–50 of 140 posts
Re: Using “underdrawings” for accurate text and numbers
#42Re: Using “underdrawings” for accurate text and numbers
#43A few months ago I tried to make Le-chat Mistral output a French poetry in Alexandrin (12 vowels). Disastrous at first. Then adding in specifications that each line had to also be transposed in IPA and each syllable counted, it went better.
Still emotionally unrelatable, but definitely was providing something that match the specifications of there are explicit and systematically enforced through deterministitic means. For now I retain that LLM limitations are thus that they can't seize the ineffable and so untrustworthy they can only be employed under very clear and inescapable constraints or they will go awry just as sure as water is wet.
Re: Using “underdrawings” for accurate text and numbers
#44I was thinking about doing the opposite for the common task of "SVG of a pelican riding a bike". Obviously, directly spitting out the SVG is gonna be bad. But image gen can produce a really stunning photorealistic image easily. Probably a good way to get an LLM to produce a decent bike-pelican SVG is to generate an image first and then get the model to trace it into an SVG. After all, few human beings can generate SVG works of art by just typing out numbers into Notepad. At the core of it, we still rely on looking at it and thinking about it as an image.
Re: Using “underdrawings” for accurate text and numbers
#45Wait, where did it get the "Sweet Path//Trail of treats" thing from in the SVG? It wasn't about sweets at that point. Something missing here, I think.
Re: Using “underdrawings” for accurate text and numbers
#46I wonder whether this could be used to fine-tune image models to provide better outputs. Something like this:
1. Algorithmically generate a underdrawing (e.g. place numbers and shapes randomly in the underdrawing)
2. Algorithmically generate a description of the underdrawing (e.g. for each shape, output text like "there is a square with the number three in the top left corner). You might fuzz this by having an LLM rewrite the descriptions in a variety of ways.
3. Generate a "ground truth" image using the underdrawing and an image+text-to-image model.
4. Use the generated description and the generated "ground truth" image as training data for a text-to-image model.
Re: Using “underdrawings” for accurate text and numbers
#47I feel sorry for the recipient.
Re: Using “underdrawings” for accurate text and numbers
#48Earlier quoted context omitted.
Nobody asked it to!
If it’s asked to generate an image, it should to everything in its powers to make the image good.
> it should do everything in its powers
That's a scary thought.
Hey Claude, why haven't you finished yet? ... Because the human I'm holding hostage hasn't finished the drawing yet.
Re: Using “underdrawings” for accurate text and numbers
#49The standard objection: if the LLM is supposedly intelligent, why can’t it figure out on its own that this two-step process would achieve a better result?
You don’t know what you don’t know
Re: Using “underdrawings” for accurate text and numbers
#50I wonder whether this could be used to fine-tune image models to provide better outputs. Something like this: 1. Algorithmically generate a underdrawing (e.g. place numbers and shapes randomly in the underdrawing) 2. Algorithmically generate a description of the underdrawing (e.g. for each shape, output text like "there is a square with the number three in the top left corner). You might fuzz this by having an LLM re…
That would complexity the architecture of a model, to solve a finite set of cases. That's an argument for specialised/fine tuned models though.