GPT-5 System Prompt?

It's because they always put things that seem way to specific to certain issues, like riddles and arithmetic. Also, I am not a WS, but the mention of "proud boys" are things that can be used as fodder for LLM bias. I wonder why they even have to use a system prompt; why can't that have a separate fine-tuned model for ChatGPT specifically so that they don't need a system prompt?

Re: GPT-5 System Prompt?

#5

Post by mdaniel » Sat, Aug 09, 2025, 4:46 PM UTC

It's a good thing people were enamored of how inexpensive GPT-5 is, given that the system prompt is (allegedly) 54kb. I don't know how many tokens that is offhand, but what a lot of them to burn just on setup of the thing

Re: GPT-5 System Prompt?

#6

Post by crazygringo » Sat, Aug 09, 2025, 4:55 PM UTC

How does a prompt this long affect resource usage?

Does inference need to process this whole thing from scratch at the start of every chat?

Or is there some way to cache the state of the LLM after processing this prompt, before the first user token is received, and every request starts from this cached state?

Re: GPT-5 System Prompt?

#7

Post by mdaniel » Sat, Aug 09, 2025, 5:02 PM UTC

crazygringo wrote: post #6

How does a prompt this long affect resource usage? Does inference need to process this whole thing from scratch at the start of every chat? Or is there some way to cache the state of the LLM after processing this prompt, before the first user token is received, and every request starts from this cached state?

My understanding is that is what the KV cache does in models serving. I would imagine they'd want to prime any such KV cache with common tokens but retain a per-session cache to avoid leaks. It seems HF agrees with the concept, at least https://huggingface.co/docs/transformers/kv_cache#prefill-a-...

Re: GPT-5 System Prompt?

#8

Post by btdmaster » Sat, Aug 09, 2025, 5:24 PM UTC

mdaniel wrote: post #5

It's a good thing people were enamored of how inexpensive GPT-5 is, given that the system prompt is (allegedly) 54kb. I don't know how many tokens that is offhand, but what a lot of them to burn just on setup of the thing

I might be wrong, but can't you checkpoint the post-system prompt model and restore from there, trading memory for compute? Or is that too much extra state?

Re: GPT-5 System Prompt?

#9

Post by mdaniel » Sat, Aug 09, 2025, 5:34 PM UTC

btdmaster wrote: post #8

mdaniel wrote: post #5

It's a good thing people were enamored of how inexpensive GPT-5 is, given that the system prompt is (allegedly) 54kb. I don't know how many tokens that is offhand, but what a lot of them to burn just on setup of the thing

I might be wrong, but can't you checkpoint the post-system prompt model and restore from there, trading memory for compute? Or is that too much extra state?

My mental model is that the system prompt isn't one thing, and that seems even more apparent with line 6 telling the model what today's date is. I have no insider information but system prompts could undergo A/B testing just like any change, to find the optimal one for some population of users

Which is to say you wouldn't want to bake such a thing too deeply into a multi-terabyte bunch of floating points because it makes operating things harder

Re: GPT-5 System Prompt?

#10

Post by Tadpole9181 » Sat, Aug 09, 2025, 6:14 PM UTC

mdaniel wrote: post #5

It's a good thing people were enamored of how inexpensive GPT-5 is, given that the system prompt is (allegedly) 54kb. I don't know how many tokens that is offhand, but what a lot of them to burn just on setup of the thing

54,000 bytes, one byte per character. 4 characters per token (more or less). Around 13,000 tokens.

These are NOT included in the model context size for pricing.