Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 System Prompt?

github.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: GPT-5 System Prompt?

#11

Post by TZubiri »

NewsaHackO wrote: post #4
TZubiri wrote: post #2

These are always so embarassing

It's because they always put things that seem way to specific to certain issues, like riddles and arithmetic. Also, I am not a WS, but the mention of "proud boys" are things that can be used as fodder for LLM bias. I wonder why they even have to use a system prompt; why can't that have a separate fine-tuned model for ChatGPT specifically so that they don't need a system prompt?

Also because we have these image of super scientist mathematician who fight for a better world and reject 1m salaries and raise billions in funding.

And their work is literally "DON'T do this, DO that in these situations"

Re: GPT-5 System Prompt?

#12

Post by reitzensteinm »

mdaniel wrote: post #9
btdmaster wrote: post #8

Earlier quoted context omitted.

I might be wrong, but can't you checkpoint the post-system prompt model and restore from there, trading memory for compute? Or is that too much extra state?

My mental model is that the system prompt isn't one thing, and that seems even more apparent with line 6 telling the model what today's date is. I have no insider information but system prompts could undergo A/B testing just like any change, to find the optimal one for some population of users Which is to say you wouldn't want to bake such a thing too deeply into a multi-terabyte bunch of floating points because it m…

OpenAI automatically caches prompt prefixes on the API. Caching an infrequently changing internally controlled system prompt is trivial by comparison.

Re: GPT-5 System Prompt?

#13

Post by kingstnap »

mdaniel wrote: post #7
crazygringo wrote: post #6

How does a prompt this long affect resource usage? Does inference need to process this whole thing from scratch at the start of every chat? Or is there some way to cache the state of the LLM after processing this prompt, before the first user token is received, and every request starts from this cached state?

My understanding is that is what the KV cache does in models serving. I would imagine they'd want to prime any such KV cache with common tokens but retain a per-session cache to avoid leaks. It seems HF agrees with the concept, at least https://huggingface.co/docs/transformers/kv_cache#prefill-a-...

OpenAI has docs about how it works.

https://platform.openai.com/docs/guides/prompt-caching

It's fairly simple actually. Each machine stores the KV cache in blocks of 128 tokens.

That's stored in a prefix tree like structure. Probably with some sort of LRU eviction policy.

If you ask a machine to generate it does so starting from the longest matching sequence in the cache.

They route between racks using a hash of the prefix.

Therefore the system prompt, being frequently used and at the beginning of the context, will always be in the prefix cache.

Re: GPT-5 System Prompt?

#14

Post by sellmesoap »

NewsaHackO wrote: post #4
TZubiri wrote: post #2

These are always so embarassing

It's because they always put things that seem way to specific to certain issues, like riddles and arithmetic. Also, I am not a WS, but the mention of "proud boys" are things that can be used as fodder for LLM bias. I wonder why they even have to use a system prompt; why can't that have a separate fine-tuned model for ChatGPT specifically so that they don't need a system prompt?

"Dear computer, I'm writing to you today to tell you to make sure you really check your math sums!" I find it amusing so much emphasis is put on a computer to get math correct.

Re: GPT-5 System Prompt?

#15

Post by crazygringo »

kingstnap wrote: post #13
mdaniel wrote: post #7

Earlier quoted context omitted.

My understanding is that is what the KV cache does in models serving. I would imagine they'd want to prime any such KV cache with common tokens but retain a per-session cache to avoid leaks. It seems HF agrees with the concept, at least https://huggingface.co/docs/transformers/kv_cache#prefill-a-...

OpenAI has docs about how it works. https://platform.openai.com/docs/guides/prompt-caching It's fairly simple actually. Each machine stores the KV cache in blocks of 128 tokens. That's stored in a prefix tree like structure. Probably with some sort of LRU eviction policy. If you ask a machine to generate it does so starting from the longest matching sequence in the cache. They route between racks using a hash of the…

Fascinating, exactly what I was wondering about. Thank you! Turns out it's very sophisticated, and also explains why the current date is always at the very end of the system prompt.

Re: GPT-5 System Prompt?

#16

Post by TZubiri »

sellmesoap wrote: post #14
NewsaHackO wrote: post #4

Earlier quoted context omitted.

It's because they always put things that seem way to specific to certain issues, like riddles and arithmetic. Also, I am not a WS, but the mention of "proud boys" are things that can be used as fodder for LLM bias. I wonder why they even have to use a system prompt; why can't that have a separate fine-tuned model for ChatGPT specifically so that they don't need a system prompt?

"Dear computer, I'm writing to you today to tell you to make sure you really check your math sums!" I find it amusing so much emphasis is put on a computer to get math correct.

And then they get offered 1M salaries for that.
Post reply on HN