Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Brex’s Prompt Engineering Guide

github.com

Post reply on HN
Print view View original

91–100 of 112 posts

Re: Brex’s Prompt Engineering Guide

#91

Post by michaelteter »

fsloth wrote: post #76
dakom wrote: post #74

Earlier quoted context omitted.

That's extremely broad. I know how to push the buttons on my washing machine. So now I'm a laundering engineer? Maybe the answer is "yes", but I think it's a bit silly, and detracts from the engineering work that goes into building the machine, i.e. the work of understanding the connections and flow of electricity from button press to pumps and gears turning.

The person who shoveled coal to a steam locomotive was called a railroad engineer. I guess the distinguishing feature here is how much capital is tied to the process and how much value it creates - not how intellectually demanding it is as such. If you had a button that switched on a thousand washing machines at a time I guess one could flirt with the title "washing engineer".

> If you had a button that switched on a thousand washing machines at a time I guess one could flirt with the title "washing engineer".

Since that button probably wouldn't work correctly 100% of the time, the operator would probably be forced to learn about problems and solutions related to that activity. Over time, that person would accumulate experience and knowledge which would make them well suited to that simple job of flipping the switch on and off. And at that point, that person would be more of an engineer than a random person who walked up and flipped the switch.

An _anything_ engineer is just someone who does that thing and the unnamed related things necessary to do the main thing.

Re: Brex’s Prompt Engineering Guide

#92

Post by abidlabs »

RC_ITR wrote: post #60
zwaps wrote: post #48

Worringly, I am it sure the people working on this really understand what a Transformer is Quote from them: “ There is still active research in non-transformer based language models though, such as Amazon’s AlexaTM 20B which outperforms GPT-3“ Quote from said paper “ For AlexaTM 20B, we used the standard Transformer model architecture“ (Its just an encoder decoder transformer)

Yeah, I think the (worrying) confusion is that Amazon calls it a seq2seq model, which was the name of a SOTA RNN from Google a while back. Ofc now, seq2seq just means what you said (an encoder/decoder model, which is actually what a “truly vanilla” transformer would be anyway). The fact that any serious researcher thinks any other serious researchers are using models without self attention is the real red flag here.…

That's actually not true: https://github.com/BlinkDL/RWKV-LM

Re: Brex’s Prompt Engineering Guide

#93

Post by abidlabs »

sgk284 wrote: post #64
zwaps wrote: post #48

Worringly, I am it sure the people working on this really understand what a Transformer is Quote from them: “ There is still active research in non-transformer based language models though, such as Amazon’s AlexaTM 20B which outperforms GPT-3“ Quote from said paper “ For AlexaTM 20B, we used the standard Transformer model architecture“ (Its just an encoder decoder transformer)

Thanks for pointing this out. That was my mistake – my brain must have swapped out "different transformer architectures" with "different model architectures". I just updated the guide: https://github.com/brexhq/prompt-engineering/commit/3a3ac17a...

A better example would have been the RWKV [1]

[1] https://github.com/BlinkDL/RWKV-LM

Re: Brex’s Prompt Engineering Guide

#94

Post by williamcotton »

dakom wrote: post #65

Why are we calling this "engineering"? Isn't engineering the application of science to solve problems? (math, definitive logic, etc.) Maybe one day we'll have instruments that let us reason about the connections between prompts and the exact state of the AI, so that we can understand the mechanics of causation, but until then, I would not think that being good at asking questions is "engineering" Are most 10 year old…

As the head of the Engineering department said the class on day one of Intro to Computer Engineering at RIT in the fall of 1999:

Engineering = Physics + Economics

The physical system here is the LLM and the computing environment that interfaces with the LLM. Prompt engineering would be the knowledge of how to use the LLM in a programmatic manner. There are many cost related trade-offs. More tokens, better response? Can you limit token usage while keeping the quality response basically the same?

Is it not obviously engineering and not just throwing stuff at the wall when you begin to measure the quality of responses with regards to token usage, finding which approaches work and which don't? Or how to handle security issues like prompt injections? Or techniques for using a vector database with latent space embeddings?

Do 10 year olds do that when they use Google?

I think the confusion here is that people think it refers to just using the ChatGPT interface and not wiring up the API to a Python interpreter.

Re: Brex’s Prompt Engineering Guide

#95

Post by eternalban »

sgk284 wrote: post #71
dakom wrote: post #65

Why are we calling this "engineering"? Isn't engineering the application of science to solve problems? (math, definitive logic, etc.) Maybe one day we'll have instruments that let us reason about the connections between prompts and the exact state of the AI, so that we can understand the mechanics of causation, but until then, I would not think that being good at asking questions is "engineering" Are most 10 year old…

Author of the guide here. I attempt to address this in the "Why do we need prompt engineering?"[^1] section. > ... we used an analogy of prompts as the “source code” that a language model “interprets”. Prompt engineering is the art of writing prompts to get the language model to do what we want it to do – just like software engineering is the art of writing source code to get computers to do what we want them to do.…

But writing “source code” is definitively not “software engineering”. It has a name: programming. A software engineer is the person that creates a system from software mostly using software. AI-assistant engineering makes sense.

Prompt crafting is fair, informative, and accurate. And we (should) all know by now that craftsman/artisan != engineer.

Ask your favorite LLM to categorize the following statements as one of { reasonable, wishful } notions:

- writing prompts for LLMs is engineering. The science of engineering effective LLM prompts.

- writing prompts for LLMs is an art*. The art of crafting effective LLM prompts.

* almost all ‘art’ rests on fairly coherent communal ‘logic’ or ‘schools’. What distinguishes art from science is that this logic does not need have any basis in appealing to reason. Aesthetics in sciences, in contrast, is primarily used to select one of many contending scientific theorems in ambiguous contexts (as in “the beauty of the mathematics of the unverifiable theory to me implies its truth”.)

Re: Brex’s Prompt Engineering Guide

#96

Post by onion2k »

fsloth wrote: post #76
dakom wrote: post #74

Earlier quoted context omitted.

That's extremely broad. I know how to push the buttons on my washing machine. So now I'm a laundering engineer? Maybe the answer is "yes", but I think it's a bit silly, and detracts from the engineering work that goes into building the machine, i.e. the work of understanding the connections and flow of electricity from button press to pumps and gears turning.

The person who shoveled coal to a steam locomotive was called a railroad engineer. I guess the distinguishing feature here is how much capital is tied to the process and how much value it creates - not how intellectually demanding it is as such. If you had a button that switched on a thousand washing machines at a time I guess one could flirt with the title "washing engineer".

The person who shoveled coal to a steam locomotive was called a railroad engineer.

Maintaining a constant head of steam while changing altitude, water level, external temperature, etc using a direct heat source is a really skilled job. Talk to anyone who works on a heritage railway and they'll explain just how hard it is.

I lived in the same city where George Stephenson invented the locomotive, and went to the same pub that the volunteers who worked on a replica went to. It's fascinating, and very much 'proper' engineering.

Re: Brex’s Prompt Engineering Guide

#97

Post by martythemaniak »

dakom wrote: post #65

Why are we calling this "engineering"? Isn't engineering the application of science to solve problems? (math, definitive logic, etc.) Maybe one day we'll have instruments that let us reason about the connections between prompts and the exact state of the AI, so that we can understand the mechanics of causation, but until then, I would not think that being good at asking questions is "engineering" Are most 10 year old…

> Isn't engineering the application of science to solve problems? (math, definitive logic, etc.)

Most definitely not. Engineering more often than not preceeds the science. You don't have to have an analytical and theoretical understanding of something in order to harness it and make practical use of it.

In reality, there's a strong feedback loop, where practical hacks will guide the science, the development of which will can then be used to make better practical applications, which can uncover new unknowns, which can then be incorporated into the science etc. The development of electricity and magnetism and or steam engines and thermodynamics are both great examples of this.

Re: Brex’s Prompt Engineering Guide

#98

Post by fsloth »

michaelteter wrote: post #91
fsloth wrote: post #76

Earlier quoted context omitted.

The person who shoveled coal to a steam locomotive was called a railroad engineer. I guess the distinguishing feature here is how much capital is tied to the process and how much value it creates - not how intellectually demanding it is as such. If you had a button that switched on a thousand washing machines at a time I guess one could flirt with the title "washing engineer".

> If you had a button that switched on a thousand washing machines at a time I guess one could flirt with the title "washing engineer". Since that button probably wouldn't work correctly 100% of the time, the operator would probably be forced to learn about problems and solutions related to that activity. Over time, that person would accumulate experience and knowledge which would make them well suited to that simple…

Exactly.

Re: Brex’s Prompt Engineering Guide

#99

Post by TrueDuality »

saladtoes wrote: post #62

I've been playing Gandalf in the last few days, it does a great job at giving an intuition for some of the subtleties of prompt engineering: https://gandalf.lakera.ai Thanks for putting this together!

Whoa that was a lot of fun. Are you aware of any other games like this? A sort of CTF for AIs?

Re: Brex’s Prompt Engineering Guide

#100

Post by fsloth »

onion2k wrote: post #96
fsloth wrote: post #76

Earlier quoted context omitted.

The person who shoveled coal to a steam locomotive was called a railroad engineer. I guess the distinguishing feature here is how much capital is tied to the process and how much value it creates - not how intellectually demanding it is as such. If you had a button that switched on a thousand washing machines at a time I guess one could flirt with the title "washing engineer".

The person who shoveled coal to a steam locomotive was called a railroad engineer. Maintaining a constant head of steam while changing altitude, water level, external temperature, etc using a direct heat source is a really skilled job. Talk to anyone who works on a heritage railway and they'll explain just how hard it is. I lived in the same city where George Stephenson invented the locomotive, and went to the same p…

"Intellectually demanding" was a poor choice of word. I meant to say something along the lines "does not require to invent anything new" - which a railroad engineer does not need to do. That does not mean work they do is low-effort or trivial.
Post reply on HN