[flagged]
The session you cannot take with you
41–50 of 238 posts
Re: The session you cannot take with you
#42OpenAI is doing this so that you can't preserve cache when routing between for example GPT 5.6 and DeepSeek V4, or other providers. I'm building a router that easily reaches >99% cache hit rate in Pi when routing between GPT 5.4, Kimi, DeepSeek etc. Working on a big update that's coming out in a week or so.
Re: The session you cannot take with you
#43> “Most People Do Not Switch Models Mid-Session”
I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models.
I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But it quickly became a habbit, to CTRL-L in pi and change the model.
Re: The session you cannot take with you
#44Re: The session you cannot take with you
#45> “Most People Do Not Switch Models Mid-Session” I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models. I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But…
Out of interest, what models do you find work best for producing text for live presentations based on the text for a reading deck?
Re: The session you cannot take with you
#46I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with. There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory n…
I'm in complete agreement that things like encrypted sub-agent messages, etc, are terrible, especially because it is fundamentally irresponsible to have an agent running on your machine without the slightest hint of what the prompt is.
That said, I don't really see a problem with hosted tools being offered by providers. They're like impulse items at checkout.
You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
Re: The session you cannot take with you
#47It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.
>and shouldn't charge more and more the longer the conversation gets The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.
Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what they are going to do as a market participant, just stating my moral preference :-).
Also, isn't it without caching even something like O(N^2) because you have to replay the whole conversation on every request to reach the same internal state? My point was I shouldn't have to pay for cache misses if hitting the cache is not deterministic. Give me a guarantee that you keep the session "hot" for N minutes, and cached on disk for M months, charge a little bit more on average, but then the pricing is at least transparent.
[1] Except the general market pressure to keep total cost for the same problem solved lower than the competition.
Re: The session you cannot take with you
#48Earlier quoted context omitted.
And yet people still go with Sign in with Google as the only login method.
I’ve actually started ignoring services where that’s the only login possibility. Sorry even if your platform is the greatest thing ever, but I’ll find a different tool. I’ve read one too many stories about Google (or Apple!) closing the entire account over some bullshit unnecessary reason like “fraudulent” gift card issues or whatever. I’m certain the affected people would’ve preferred to just pay back the amount in…
It's the only reason I want to replace Tailscale with something else or look into self-host when have a bit of time during my vacation. They only allow login through a third party, which is a big no for me.
Re: The session you cannot take with you
#49OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible. If you have patience and the willingness to endure a little bit o…
You don't see real reasoning tokens with the Chat Completions API; and 'store: false' with Responses really isn't that different to Completions.
FWIW, if you have some tokens to spend, you might want to test Responses vs Completions in intelligence. Since GPT-5 models, we've consistently seen small, but statistically significant and reproducible improvements in intelligence with Responses API vs Completions.
However it works underneath the hood, it's real.
Re: The session you cannot take with you
#50I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with. There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory n…
I'm in complete agreement that things like encrypted sub-agent messages, etc, are terrible, especially because it is fundamentally irresponsible to have an agent running on your machine without the slightest hint of what the prompt is. That said, I don't really see a problem with hosted tools being offered by providers. They're like impulse items at checkout. You shouldn't implement image generation as MCP: just writ…
Hosted tools are absolutely fine. Hiding away their results in encrypted blobs less so.