Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The session you cannot take with you

earendil.com

Post reply on HN
Print view View original

31–40 of 238 posts

Re: The session you cannot take with you

#31

Post by the_mitsuhiko »

swyx wrote: post #8
mnewme wrote: post #6

Thank you Armin, Mario and team for the great writeup, absolutely agree and I love what you guys are building

what would it take for this to be more than just hn blog fodder?

We were thinking about it while writing it. I think one thing at least open source harnesses can do is push back on using some of these conveniences from the providers. For instance the moment we start compromising on the idea of portable sessions, then there will be less pressure (For instance people asked for OpenAI server side compaction or built-in native web_search support).

The real thing that will take this alive though is people pushing back a bit against some of these newfangled APIs. Now that there is real competition from the new generation of Chinese models which have much fewer of those restrictions, I think there might be a moment.

Re: The session you cannot take with you

#33

Post by skybrian »

I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good.

In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.

Re: The session you cannot take with you

#34

Post by charcircuit »

bob1029 wrote: post #28

OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible. If you have patience and the willingness to endure a little bit o…

OpenAI has hidden its reasoning for almost 2 years.

Re: The session you cannot take with you

#35

Post by hobofan »

n0on3 wrote: post #3

I get the point of the article and it resonates with my general tendency to support more openness, but older and more cynical me wonders how much of this is a battle worth fighting or a self-serving argument from an harness developer. I mean, the article says each of the points it is complaining about has “_a basic justification that's trivial for a provider to come up with, along with good arguments for why this is…

> these are just bs to have people accept them, but is that actually the case?

I tried to outline that in my sibling comment[0], but I think the article also gives a good example in the "hidden searches" paragraph: There is really no good reason not to expose details about many of the builtin tools, or provide them as standalone services, other than protect their moat.

I don't think that "role confusion" attacks could be meaningful prevented by not showing reasoning traces. Prevention of reasoning traces is about as futile as prevention of system prompt leakage. And here you would just have to manage to exfiltrate a small number of reasoning traces per model family in order to distill the writing style of thinking traces.

[0]: https://news.ycombinator.com/item?id=49120015

Re: The session you cannot take with you

#36

Post by mystifyingpoi »

eru wrote: post #16
qurren wrote: post #14

Earlier quoted context omitted.

Because with the alternative they'll require you to "confirm your e-mail" instead of just picking a password

I'm actually more annoyed by having to pick passwords than confirming my email.

I'm opposite. If my email goes down, or I do not have access to it, I cannot login. With passwords I still can.

Re: The session you cannot take with you

#38

Post by hobofan »

bob1029 wrote: post #28

OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible. If you have patience and the willingness to endure a little bit o…

> Chat completion can still do everything if you are clever.

On a purely functional level, yes. However for interactive use cases, the Completions API, as provided by OpenAI or Azure, if paired with reasoning effort of any kind, provides an awful user experience, as you will have a perceived delay of 10+ seconds until the first tokens stream in.

If using other providers that are exposing their thinking traces, this is less of an issue, as they've just extended the Comletions API format to have delta events with reasoning_content.

Re: The session you cannot take with you

#39

Post by djfergus »

jorisw wrote: post #19
Centigonal wrote: post #5

I think a fantastic companion piece to this article is https://gwern.net/complement

Say why

“A classic pattern in technology economics, identified by Joel Spolsky, is layers of the stack attempting to become monopolies while turning other layers into perfectly-competitive markets which are commoditized, in order to harvest most of the consumer surplus; discussion and examples”

Re: The session you cannot take with you

#40

Post by captainmuon »

the_mitsuhiko wrote: post #30
captainmuon wrote: post #4

It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.

Because if you don't, you trash the KV cache and things get way more expensive. We wrote about it here: https://earendil.com/posts/prompt-caching/ Encrypted reasoning content is one of the things we keep as blobs in the transcript, even though we can only send it back to the original provider that created them, as otherwise we cannot keep caches live.

Right, but my point was, I should be able to just send something like

    {'continue_conversation_id': 123, 'message': 'another message?'}
Seems like it would be even less work for the LLM host to just check if ID 123 is still in a cache (and if not, load it from a database) than decoding my request and checking cryptographic signatures. Right now the whole conversation history with blobs functions like a really long ID.
Post reply on HN