21–30 of 33 posts
requires a backwards trained llm no? I don't think anyone has pretrained a remotely-close-to-SOTA sized backwards model.
We are continuously adding more benchmarks to the paper with UTAustin.
> - the task of crafting textual inputs to effectively direct LLMs -- remains difficult and labor-intensive Damn, I knew we were lazy but describing prompting as labor-intensive is impressively lazy even to me. Obviously reading the rest of the abstract was too labor intensive for me but I'm hoping I can just hook a probe up to my drool and it can infer my desires from that.
Setting parameters for any ml model is easy, but we'd call it labour intensive if we expected people to do it manually despite having evals. Instead we have ways of searching for and optimising settings. The methods for that are obvious for small cardinality discrete values or continuous variables. Less so for arbitrary text.
I always find myself baffled by “prompt optimization” frameworks. Do people really find themselves needing random perturbations of a fixed prompt to improve accuracy? It’s my experience that the challenging part of writing a prompt is figuring out what the task you want done is, and understanding which data you need to pass to the model to make the task achievable. None of that can be achieved by “optimizing” the pro…
I always find myself baffled by “prompt optimization” frameworks. Do people really find themselves needing random perturbations of a fixed prompt to improve accuracy? It’s my experience that the challenging part of writing a prompt is figuring out what the task you want done is, and understanding which data you need to pass to the model to make the task achievable. None of that can be achieved by “optimizing” the pro…
> - the task of crafting textual inputs to effectively direct LLMs -- remains difficult and labor-intensive Damn, I knew we were lazy but describing prompting as labor-intensive is impressively lazy even to me. Obviously reading the rest of the abstract was too labor intensive for me but I'm hoping I can just hook a probe up to my drool and it can infer my desires from that.
This must be what AI hype actually is. Complete incoherent language to explain a very straight forward concept.
This is just: LLMs judging intermediate node outputs, and reverse traversing the graph while doing so until it modifies the original prompt.
Holy unnecessary use of terminology to explain a reverse graph traversal. “Loss”, “gradients”, “differentiating”— no! stop! This must be what AI hype actually is. Complete incoherent language to explain a very straight forward concept. This is just: LLMs judging intermediate node outputs, and reverse traversing the graph while doing so until it modifies the original prompt.
I always find myself baffled by “prompt optimization” frameworks. Do people really find themselves needing random perturbations of a fixed prompt to improve accuracy? It’s my experience that the challenging part of writing a prompt is figuring out what the task you want done is, and understanding which data you need to pass to the model to make the task achievable. None of that can be achieved by “optimizing” the pro…
depends what you're doing. If you're using ChatGPT via the UI for a one off question, sure. If you're prompting an LLM that is doing a critical task in production millions of times, minor improvements can have significant benefit
To the extent that you need to eke out reliability on the margins, one is vastly better served by actual fine-tuning, which is available both for open-source models and most major proprietary models.
Holy unnecessary use of terminology to explain a reverse graph traversal. “Loss”, “gradients”, “differentiating”— no! stop! This must be what AI hype actually is. Complete incoherent language to explain a very straight forward concept. This is just: LLMs judging intermediate node outputs, and reverse traversing the graph while doing so until it modifies the original prompt.
i am old enough to remember the opposite: people would try to sell deep learning to the mainstream ML community by pointing out that backprop is just message-passing on a Bayesian network with modified sum/product operations.