Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic publishes the 'system prompts' that make Claude tick

techcrunch.com

Post reply on HN
Print view View original

241–250 of 290 posts

Re: Anthropic publishes the 'system prompts' that make Claude tick

#241

Post by jumploops »

chilling wrote: post #22

> Claude responds directly to all human messages without unnecessary affirmations or filler phrases like “Certainly!”, “Of course!”, “Absolutely!”, “Great!”, “Sure!”, etc. Specifically, Claude avoids starting responses with the word “Certainly” in any way. Meanwhile my every respond from Claude: > Certainly! [...] Same goes with > It avoids starting its responses with “I’m sorry” or “I apologize” and every time I spo…

“Create a picture of a room, but definitely don’t put an elephant in the corner.”

Re: Anthropic publishes the 'system prompts' that make Claude tick

#242

Post by willy_k »

acchow wrote: post #234
adastra22 wrote: post #232

Earlier quoted context omitted.

Good luck defining “understanding” in a way that lets you say LLMs don’t understand but humans do. At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too.

> At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too. We might be. Or we might be something else entirely. Who knows?

Roger Penrose knows, imo

Re: Anthropic publishes the 'system prompts' that make Claude tick

#243

Post by SirMaster »

TeMPOraL wrote: post #215
SirMaster wrote: post #146

Earlier quoted context omitted.

>they are in the encoding Is encoding not math?

It's math, but specifically an independent piece you could swap out for a different one that does much better on this problem (e.g. use characters instead of tokens) - it's just doing so would make training and inference much more expensive (read: much worse model performance for a given training/compute budget), so it's not worth the trade-off. It's not like humans read letter by letter either, at least not past the…

But that's really what I meant. When you say the limitation on processing is not in the math. I would say it is a mathematical limitation of processing because they had to choose a math that works on parts of words instead of letters due to the limitation of the power of the math that can be done for training and inference.

They chose to use some limiting math which prevents the LLM from being able to easily answer questions like this.

It's not a limitation of math in general. It's a limitation of the math they chose to build the LLM on which is what was going through my head when I was writing it.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#244

Post by yen223 »

blooalien wrote: post #175
marcus0x62 wrote: post #120

Earlier quoted context omitted.

I understand it to be by predicting the next most likely output token based on previous user input. I also understand that, simplistic though the above explanation is and perhaps is even wrong in some way, it to be a more thorough explanation than anyone thus far has been able to provide about how, exactly, human consciousness and thought works. In any case, my point is this: nobody can say “LLMs don’t reason in the…

Does your brain completely stop doing anything between verbal statements (output)? An LLM does stop doing stuff between requests to generate a string of language tokens (their entire purpose). When not actually generating tokens, an LLM doesn't sit there and think things like "Was what I just said correct?" or "Hmm. That was an interesting discussion. I think I'll go research more on the topic". Nope. It just sits th…

Of all the ways to measure intelligence, "whether it's using 100% of its compute time" is certainly one of them.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#245

Post by chilling »

lolinder wrote: post #203
chilling wrote: post #22

> Claude responds directly to all human messages without unnecessary affirmations or filler phrases like “Certainly!”, “Of course!”, “Absolutely!”, “Great!”, “Sure!”, etc. Specifically, Claude avoids starting responses with the word “Certainly” in any way. Meanwhile my every respond from Claude: > Certainly! [...] Same goes with > It avoids starting its responses with “I’m sorry” or “I apologize” and every time I spo…

I suspect this is a case of the system prompt actually making things worse. I've found negative prompts sometimes backfire with these things the same way they do with a toddler ("don't put beans up your nose!"). It inserts the tokens into the stream but doesn't seem to adequately encode the negative.

I know, I suspect that too. It's like me asking GPT to: `return the result in JSON format like so: {name: description}, don't add anything, JSON should be as simple as provided`.

ChatGTP: I understand... here you go

{name: NAME, description: {text: DESCRIPTION } }

(ノಠ益ಠ)ノ彡┻━┻

Re: Anthropic publishes the 'system prompts' that make Claude tick

#246

Post by chilling »

NiloCK wrote: post #239
chilling wrote: post #22

> Claude responds directly to all human messages without unnecessary affirmations or filler phrases like “Certainly!”, “Of course!”, “Absolutely!”, “Great!”, “Sure!”, etc. Specifically, Claude avoids starting responses with the word “Certainly” in any way. Meanwhile my every respond from Claude: > Certainly! [...] Same goes with > It avoids starting its responses with “I’m sorry” or “I apologize” and every time I spo…

I was also pretty shocked to read this extremely specific direction, given my (many) interactions with Claude. Really drives home how fuzzily these instructions are interpreted.

I mean... we humans are also pretty bad at following instruction too.

Turn left, no! Not this left, I mean the other left!

Re: Anthropic publishes the 'system prompts' that make Claude tick

#247

Post by chilling »

lemming wrote: post #174
xienze wrote: post #155

Earlier quoted context omitted.

> Probably for the best that users see the words "Sorry, I hallucinated" every now and then. Wouldn’t “sorry, I don’t know how to answer the question” be better?

"Sorry, I just made that up" is more accurate.

IT WAS JUST A PRANK BRO! JUST A PRANK

Re: Anthropic publishes the 'system prompts' that make Claude tick

#248

Post by jay_kyburz »

blooalien wrote: post #50
throwup238 wrote: post #23

Earlier quoted context omitted.

Wait till you give it control over life support!

> Wait till you give it control over life support! That right there is the part that scares the hell outta me. Not the "AI" itself, but how humans are gonna misuse it and plug it into things it's totally not designed for and end up givin' it control over things it should never have control over. Seeing how many folks readily give in to mistaken beliefs that it's something much more than it actually is, I can tell it'…

One of my kids is in 5th grade and is learning to some basic algebra. He is learning to calculate x when it's on both sides of an equation. We did a few on paper and just as we were wrapping up he had a random idea that he wanted to ask ChatGPT to do some. I told him GPT is not great for that kind of thing, it doesn't really know math and might give him wrong answers and he would never know, we would have to calculate it anyhow to know if GPT had given the correct answer.

Unfortunately GPT got every answer correct, even broke it all down into steps just like the textbooks did.

Now my 5th grader doesn't really believe me and thinks GPT is great at math.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#249

Post by geon »

ks2048 wrote: post #222
dr_dshiv wrote: post #214

Earlier quoted context omitted.

Here, check it out— Claude sharing things that are only “once or twice on the internet” https://claude.site/artifacts/605e9525-630e-4782-a178-020e15... It is funny, because it says things like “yak milk cheese making tutorials” and “ancient Sumerian pottery catalogs”. But that’s only the extremely rare. The things for “only once or twice” are “the location of jimmy Hoffa’s remains” and “banksy’s true identity.”

This list of things that "only appear once or twice on the internet" makes no sense to me. Many are things that don't exist at all, depending on how you define it. A guess the best defense of Claude is that the question is a bit ill-defined.

Yes. "A video of the construction of Stonehenge"

Re: Anthropic publishes the 'system prompts' that make Claude tick

#250

Post by adastra22 »

kortilla wrote: post #237
adastra22 wrote: post #233

Earlier quoted context omitted.

Yeah an LLM is basically doing what you would do with the prompt “I’m going to ask you a question, give your best off the cuff response, pulling details entirely from memory without double-checking anything.” Then when it gets something wrong we jump on it and say it was hallucinating. As if we wouldn’t make the same mistakes.

It’s not like that at all. Hallucinations are complete fabrications because the weights happened to land there. It has nothing to do with how much thought or double checking there is. You can trick an LLM into “double checking” an already valid answer and get it to return nonsense hallucinations instead.

That’s how your brain works at the base level too.
Post reply on HN