Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic publishes the 'system prompts' that make Claude tick

techcrunch.com

Post reply on HN
Print view View original

281–290 of 290 posts

Re: Anthropic publishes the 'system prompts' that make Claude tick

#281

Post by fennecfoxy »

throwway_278314 wrote: post #100
mrfinn wrote: post #87

they’re simply statistical systems predicting the likeliest next words in a sentence They are far from "simply", as for that "miracle" to happen (we still don't understand why this approach works so well I think as we don't really understand the model data) they have a HUGE amount relationships processed in their data, and AFAIK for each token ALL the available relationships need to be processed, so the importance of…

> And I fail to see why our human brains couldn't be doing something very, very similar with our language capability. Then you might want to read Cormac McCarthy's The Kekulé Problem https://nautil.us/the-kekul-problem-236574/ I'm not saying he is right, but he does point to a plausible reason why our human brains may be doing something very, very different.

Onus of proof fallacy (basically "find the idea I'm referring to yourself"). You might want to clarify or distill your point from that publication without requiring someone to read through it.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#282

Post by adastra22 »

dlenski wrote: post #273
adastra22 wrote: post #272

Earlier quoted context omitted.

There is no "mysterious, emergently random force that determines what path a given quantum system will take." This is complete BS. Sorry to pull an argument from authority, but I am a trained physicist. This is a persistent misunderstanding of entanglement and so-called "collapse" of the wave function, dating back to a popular science misunderstanding of an incomplete and since discredited interpretation of quantum m…

> There is no guiding hand, metaphorical or literal, choosing how a quantum system evolves. Indeed, nicely put. To be even more specific about why not: Bell's theorem ( https://en.wikipedia.org/wiki/Bell%27s_theorem ) shows that, with some reasonable assumptions about locality, quantum mechanics cannot be explained away by a set of hidden variables that guide an "underlying" deterministic/non-random system.

I think what you're saying might be construed to be the opposite of what you're intending, so just to clarify: Bell's inequality implies that IF there is some sort of underlying force guiding quantum phenomena, then it must be non-local (AKA faster-than-light). For deep technical reasons this is such a hard pill to swallow, that a physicist would choose almost any other theory over this. It's effectively positing an infinite, all-knowing god, as anything less than this would not be able to consistently control selection of these quantum choices.

It's an added reason to be dubious though. The primary and most fundamental reason to reject this idea of "quantum selection" is that nothing is actually being selected. In a system with two possible outcomes, both happen. "We" (the current in-this-moment "we") end up in one of those paths with some probability, but both outcomes actually do happen. This is the standard, accepted model of physics today.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#283

Post by dizhn »

amanzi wrote: post #35
atorodius wrote: post #3

Personally still amazed that we live in a time where we can tell a computer system in pure text how it should behave and it _kinda_ works

I was just thinking the same thing. Usually programming is a very binary thing - you tell the computer exactly what to do, and it will do exactly what you asked for whether it's right or wrong. These system prompts feel like us humans are trying really hard to influence how the LLM behaves, but we have no idea if it's going to work or not.

We also have no idea HOW it works. We're trying to poke it with a stick from afar in order to make it do what we want.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#284

Post by dizhn »

CamperBob2 wrote: post #129
SirMaster wrote: post #61

Earlier quoted context omitted.

If it truly understood what things mean, then it would be able to tell me how many r's are in the word strawberry. But it messes something so simple up because it doesn't actually understand things. It's just doing math, and the math has holes and limitations in how it works that causes simple errors like this. If it was truly understanding, then it should be able to understand and figure out how to work around these…

If it truly understood what things mean, then it would be able to tell me how many r's are in the word strawberry. How about if it recognized its limitations with regard to introspecting its tokenization process, and wrote and ran a Python program to count the r's? Would that change your opinion? Why or why not?

Would it even know when to resort to this ?

Re: Anthropic publishes the 'system prompts' that make Claude tick

#285

Post by valval »

criddell wrote: post #78
Terr_ wrote: post #29

Earlier quoted context omitted.

LLM Prompt Engineering: Injecting your own arbitrary data into a what is ultimately an undifferentiated input stream of word-tokens from no particular source, hoping your sequence will be most influential in the dream-generator output, compared to a sequence placed there by another person, or a sequence that they indirectly caused the system to emit that then got injected back into itself. Then play whack-a-mole unti…

It probably shouldn't be called prompt engineering , even informally. The work of an engineer shouldn't require hope .

What a silly thing to say. Engineering is just problem solving.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#286

Post by fnordpiglet »

adastra22 wrote: post #232
fnordpiglet wrote: post #225

Earlier quoted context omitted.

I’d also note this isn’t confidence in the answer but in the token prediction. As LLMs have no conceptual “understanding,” they likewise have no computable confidence in accuracy of the correctness of their answers as we understand correctness. While certainly token confidence can be a proxy it’s not a substitute.

Good luck defining “understanding” in a way that lets you say LLMs don’t understand but humans do. At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too.

I didn’t say humans can compute how likely their answers are. I said LLM token prediction scores aren’t confidence of accuracy but likelihood of the tokens given the chain of prior token. The assertion that it is some sort of “accuracy” metric is false. The humans can’t either is a whatsboutism

Re: Anthropic publishes the 'system prompts' that make Claude tick

#287

Post by fnordpiglet »

viraptor wrote: post #236
fnordpiglet wrote: post #225

Earlier quoted context omitted.

I’d also note this isn’t confidence in the answer but in the token prediction. As LLMs have no conceptual “understanding,” they likewise have no computable confidence in accuracy of the correctness of their answers as we understand correctness. While certainly token confidence can be a proxy it’s not a substitute.

> I’d also note this isn’t confidence in the answer but in the token prediction. I really don't understand the distinction you're trying to make here. Nor how do you define "computable confidence" - when you ask an LLM to give you a confidence value, it is indeed computed. (It may not be the value you want, but... it exists)

The assertion that the token likelihood metric is some sort of accuracy metric is false. There are more traditional AI techniques that compute probabilistic reasoning scores that are in fact likelihoods of accuracy.

I’d note you can’t ask an LLM for a confidence value and get any answer that’s not total nonsense. The likelihood scores for the token prediction given prior tokens isn’t directly accessible to the LLM and isn’t intrinsically meaningful regardless in the way people hope it might be. They can quite confidentially produce nonsense with a high likelihood score.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#288

Post by brookst »

SirMaster wrote: post #243
TeMPOraL wrote: post #215

Earlier quoted context omitted.

It's math, but specifically an independent piece you could swap out for a different one that does much better on this problem (e.g. use characters instead of tokens) - it's just doing so would make training and inference much more expensive (read: much worse model performance for a given training/compute budget), so it's not worth the trade-off. It's not like humans read letter by letter either, at least not past the…

But that's really what I meant. When you say the limitation on processing is not in the math. I would say it is a mathematical limitation of processing because they had to choose a math that works on parts of words instead of letters due to the limitation of the power of the math that can be done for training and inference. They chose to use some limiting math which prevents the LLM from being able to easily answer q…

The LLM only sees tokens. The limitation is in the E2E product because of the encoder chosen. Change the encoder, keep the LLM, different limitations appear.

Perhaps it’s a pedantic difference, but to someone in the field the complaint reads like saying TCP/IP is deficient because it doesn’t support encryption: technically true but missing context about the whole stack.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#289

Post by brookst »

Terr_ wrote: post #196
brookst wrote: post #82

Earlier quoted context omitted.

Wholeheartedly agree. Which is why the most valuable people in a company are those who can cross abstraction layers, vertically or horizontally, and reduce information loss from boundaries between abstractions.

Some executive: "That's nice, but what new feature have you shipped for me recently?"

Hey, you read my review!

Re: Anthropic publishes the 'system prompts' that make Claude tick

#290

Post by generalizations »

SparkyMcUnicorn wrote: post #267
generalizations wrote: post #206

Earlier quoted context omitted.

Weird. I actually used the same prompt with both, just swapped out the model API. Used python because GPT4 seemed to gravitate towards it. I wonder if OpenAI tried for newer training data? Maybe Sonnet 3.5 just hasn't seen enough recent rust code. Also curious, I run into trouble when the output program is >8000 tokens on Sonnet. Did you ever find a way around that?

Sonnet 3.5 has a max output of 8192 tokens[0]. I break most tasks down into parts. Aider[1] is essential to my workflow and helps with this as well, and it's a fantastic tool to learn from. In fact, as of v0.52 I'm able to remove some of my custom code to run and test. Started playing around with adding Nous[2] as well (aider is its code editing agent), but not enough that I'm using it practically yet. [0] https://do…

Yeah, I know about the max tokens. As long as the code stays below that limit, I can get sonnet to emit complete python scripts; run those directly & return the results to sonnet, and I have a great feedback loop. That breaks down when sonnet can't emit all the code at once, because then you have to figure out ways to predictably assemble a larger script from a smaller one...that's where I couldn't find a decent solution.
Post reply on HN