Ask HN: Is prompt engineering just snake oil?

Is prompt engineering a snake oil? The engineering, as the word stands so far in history is about observing clear limitations and capabilities of something and then "engineering" things around it such as "engineering a compiler" is all about knowing the underlying processor, memory and other characteristics and then engineering a solution that converts a text notation to a stream of instructions that try to strike a balanced tradeoff.

But with LLMs, no one knows their workings once they are trained. The other day, Bark model[0] for text to speech, the team itself has following to say on details:

> Below is a list of some known non-speech sounds, but we are finding more every day. Please let us know if you find patterns that work particularly well on Discord! [laughter], [laughs], [sighs]....

So that's the team themselves not knowing what their model is capable of then how come prompt engineering is any engineering at all?

[0] https://github.com/suno-ai/bark

Yes

In general, any product or service that makes exaggerated or false claims of effectiveness without providing concrete evidence to support those claims could be considered a form of "snake oil."

Just because we don't know all the workings of something doesn't mean we can't effectively use it, or measure how well that approach of use is valuable or repeatable for similar uses.

For example, while we have a good understanding of the basic principles of magnetism, there is still much that we do not fully understand about how magnets work, and ongoing research is focused on unraveling these mysteries and expanding our understanding of this fascinating natural phenomenon.

That's not to say we don't know how to use magnets to make sound, however.

Prompt engineering, in my opinion, is a process of optimized querying of a frozen model. The approach can be augmented by using "hot" data from vector databases or other types of text storage engines. The approach to picking the right content for prompt building is not "snake oil", but based on well known processes including autocomplete, synonyms and related terms, personalization, and query expansion.

Think of it more as "prompt-fu", akin to "google-fu"--or how to google. You do have to know how to talk to an LLM, and it will be important. But it's not so much engineering, as we come to think of it in software traditionally. I wouldn't get hung up on the name, but just see it for what it is--a bad name for something we'll need to learn to be effective. It won't go away entirely, but will be mitigated to be less important as LLMs and AI get better.