Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The session you cannot take with you

earendil.com

Post reply on HN
Print view View original

101–110 of 238 posts

Re: The session you cannot take with you

#101

Post by charcircuit »

captainmuon wrote: post #47
charcircuit wrote: post #7

Earlier quoted context omitted.

>and shouldn't charge more and more the longer the conversation gets The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.

Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what…

Ultimately you are paying for quality so you end up relying on [1] regardless. Even if they show you tokens they can cheat by using weaker models, showing fake thinking tokens, etc.

>because you have to replay the whole conversation on every request to reach the same internal state?

It's because you have to rebuild what would have been cached for every token before the latest one that is being worked on.

Re: The session you cannot take with you

#102

Post by the_mitsuhiko »

agilek wrote: post #10
solarkraft wrote: post #9

This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ... > Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you. This is why it’s important to utilize your freedoms. Do NOT…

And yet people still go with Sign in with Google as the only login method.

I guess the good aspect of sign in with Google is that it falls back nicely to sign-in via email which is very portable.

Re: The session you cannot take with you

#103

Post by pshirshov »

I have some sort of a "solution" - my process keeps state that really matters (think "JIRA for agents") in a separate database (accessible over MCP). Essentially, I can terminate my session, start fresh one (in a different harness with a different model) and continue work with minimal losses. For subagents - I have a custom dispatch_agent tool which, essentially, just shells out.

Re: The session you cannot take with you

#104

Post by lukebuehler »

the_mitsuhiko wrote: post #95
lukebuehler wrote: post #91

Earlier quoted context omitted.

Fair. But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. There is just a whole discussion to be had about the second issue and the two are connected. For example session caching--and similar features in the future--will introduce session incompatibilities that break portability too.

> But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. But not on the level that providers need to agree to a common format. Within Pi we are able to model portable sessions between providers even if they have provider specific APIs. For instance most models do not agree on how to do deferred tool loading. But…

Yes, I agree with your point and article regarding hidden or sealed state.

So, I'm aware that this is a separate point from the article, and I'm well aware of Pi's model abstraction layer, which is one of the best (others are a big pain... looking at you LangChain). I've come to the conclusion that full session portability will come to an end very soon, and really already has. Partially because of the hidden state stuff, partially because of feature divergence. We can still kinda patch over it right now, but it's getting harder by the week. Some examples: computer use structures in OAI responses, until recently MCP tunnels were only supported by OAI, not Anthropic, and tool discovery/search API is also getting very difficult to model fully in a unified abstraction (still possible as you point out), automatic compaction is also getting gnarly, etc.

Re: The session you cannot take with you

#105

Post by benob »

solarkraft wrote: post #9

This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ... > Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you. This is why it’s important to utilize your freedoms. Do NOT…

People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277, in addition to facilitating distillation (you don't pay the full cost of RL)

Re: The session you cannot take with you

#106

Post by springtimesun »

cube00 wrote: post #76
skybrian wrote: post #33

I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good. In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.

> Conversations contain a bunch of junk anyway This is what brought me around to doing more agentic coding. I set the task, require tests and the strict linting must pass and then leave it to blow smoke up its own ass about what's going on. I see glimpses scrolling past of all the conversational language that used to frustrate me so much when using a chat interface and I can just let it flow on past. I come back when…

I have been doing this with opencode. Spend a fair bit of time composing and scoping a ticket. tmux on the server where the local inference lives, start a non-interactive session that starts with “read issue #123, do all your work in a feature branch” and ends with “push and open a PR”. Like not watching someone else load your dishwasher, you appreciate the end result a lot more if you don’t watch the process, even if you rearrange things before you start it.

Re: The session you cannot take with you

#107

Post by the_mitsuhiko »

lukebuehler wrote: post #104
the_mitsuhiko wrote: post #95

Earlier quoted context omitted.

> But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. But not on the level that providers need to agree to a common format. Within Pi we are able to model portable sessions between providers even if they have provider specific APIs. For instance most models do not agree on how to do deferred tool loading. But…

Yes, I agree with your point and article regarding hidden or sealed state. So, I'm aware that this is a separate point from the article, and I'm well aware of Pi's model abstraction layer, which is one of the best (others are a big pain... looking at you LangChain). I've come to the conclusion that full session portability will come to an end very soon, and really already has. Partially because of the hidden state st…

It definitely is getting quite challenging. That said, a lot of user experiences are modelled around retaining transcripts for longer than a few minutes. Every claw or chat based application creates a very real user experience desire of that conversation being something you can continue, even weeks after when the model might have gone away / have been replaced.

Now unless you fully want to commit yourself to one provider, you will on the application layer need to deal with this anyways in one form or another. We want to at least try to address most of this on the harness layer, regardless of how tricky that is.

Re: The session you cannot take with you

#108

Post by springtimesun »

the_mitsuhiko wrote: post #31
swyx wrote: post #8

Earlier quoted context omitted.

what would it take for this to be more than just hn blog fodder?

We were thinking about it while writing it. I think one thing at least open source harnesses can do is push back on using some of these conveniences from the providers. For instance the moment we start compromising on the idea of portable sessions, then there will be less pressure (For instance people asked for OpenAI server side compaction or built-in native web_search support). The real thing that will take this al…

I hadn’t really thought about it until I read this (and happen to be spending time reading Kimi’s reasoning), but I can imagine an issue in the future where I will want more than just git blame + the ticket + the PR. Attaching an entire session in full to the PR could be useful archeology. And agents can trivially take it all into context. Maybe that actually turns into a competitive advantage for the open models.

Re: The session you cannot take with you

#109

Post by falcor84 »

pcthrowaway wrote: post #85
blauditore wrote: post #56

Earlier quoted context omitted.

> [...] dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good This is definitely not the way most software has been going in the past decades. It's rather the opposite: Companies play friendly to obtain a user base, then start applying more and more dark patters to further increase their pr…

Also, see Android and iOS which have a monopoly on the mobile OS market

Sorry about the nitpicking, but s/monopoly/duopoly/

Re: The session you cannot take with you

#110

Post by zer00eyz »

throwaway63467 wrote: post #11

I think most LLM work will migrate to open weight models in a couple of years for this exact reason (as well as price), unless the big providers manage to have their way and the market gets regulated in a way that favors closed source models. To me it seems though like most models are becoming good enough for most tasks we want them to do so no need to use the latest super intelligent one that knows how to compose op…

> (as well as price)

LLM pricing is now Business Gacha - the whales will open more loot boxes and the normals will serve as fodder and feeder for them.

> market gets regulated

This MUST happen, but not in the way or for the reasons that any one thinks. There is a question of liability when one of these models causes massive damage to a 3rd party.

If I am running an open source model on a "rented" platform and it goes off the rails who is to blame when it "breaks containment" and does something bad?

The liability people (read lawyers) are gonna figure this out a lot faster than any one who says the word "safety" a lot.

Post reply on HN