All of this can be addressed by using locally hosted models running in bespoke harness rigs.
The session you cannot take with you
161–170 of 238 posts
Re: The session you cannot take with you
#162Earlier quoted context omitted.
> Conversations contain a bunch of junk anyway This is what brought me around to doing more agentic coding. I set the task, require tests and the strict linting must pass and then leave it to blow smoke up its own ass about what's going on. I see glimpses scrolling past of all the conversational language that used to frustrate me so much when using a chat interface and I can just let it flow on past. I come back when…
Maybe I am still a novice. When I insist on tests, I get plenty of them, but on close review I'll ask "did you handle [obvious and important thing]" and suddenly it's tearing the code apart and building more stuff. It just loves corner cutting.
> Maybe I am still a novice.
With Boris telling everyone they should throw away their AGENTS.md file this week, I think we're all back to being novices again.
> It just loves corner cutting.
Absolutely, it tried to disable a bunch of lint rules for me calling them "overly stylistic". It also casually dropped some unsafe Rust blocks in an run of the mill CRUD application that definitely had no business needing unsafe.
No, sorry, not good enough, go back to talking to yourself until its done properly.
It eventually got there taking most of my quota with it. I'm hoping once sandboxes is more fine grained we'll be able to lock out configuration files to prevent any edits on them.
Re: The session you cannot take with you
#163This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ... > Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you. This is why it’s important to utilize your freedoms. Do NOT…
People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277 , in addition to facilitating distillation (you don't pay the full cost of RL)
The latter ("facilitating distillation") is a real concern from the labs I'm sure, but the first part I don't understand what you mean, or you misunderstand that paper, it's about "prompt injection" via manipulating the reasoning, not about the model reasoning by itself and somehow that leading to more prompt injections. They're quite literally maliciously rewriting the reasoning as the model reasons, not just showing it to the end-user, two very different things.
Re: The session you cannot take with you
#164It's important to realize that the session is jointly owned between the model and the user. Part of the session, such as the reasoning tokens, are generated entirely by the model. Other parts, entirely by the user. However, the ownership and use of all parts is subject to the service provider's terms and conditions, which the user accepts prior to access to the service.
I don't use these models but I am confident the T&Cs establish the service provider's rights in all of the bullets mentioned, from what can be done with the user's prompts to how searches and reasoning contexts are managed. In that sense, it's very similar to cloud services, and we see an overlap between service providers in both sectors.
Ultimately one is buying a service, and all boundaries around what can and cannot be done are defined by the terms and conditions. If portability is a hard requirement, then the best thing to do is to look for services that enshrine portability in their terms.
Re: The session you cannot take with you
#165Maybe we don't take sessions, but take something else that we can fully control. Sessions were always a poor substitute to capture or archive the basis on which the implemented code rested on. They get noisy over time even with compaction, were not always inspectable, no real science behind when to ditch a session and start a new one, and rarely have a clean mapping to the work done. So instead of trying to archive sessions as a means to preserve basis, we should create our own structures to make the agent/model capture basis outside of a session, and archive that instead.
Re: The session you cannot take with you
#166Amazing article that puts my outsider observations into technically-digestible details. Tangentially, their repeated observations to providers hiding or obfuscating important systems or processes are highly relevant to my own developing case studies for my CurrentCo.
The problem with building automations atop these systems isn’t entirely their probabilistic nature (though that is the lion’s share, at least for me personally), but also the inability to effectively troubleshoot the processes themselves due to key components being obfuscated from view. How can we effectively troubleshoot what went wrong in an agentic loop when we cannot see the reasoning tokens generated from our inputs? How can we triage a broken process when token logs aren’t ours to view? How does one create determinism from increasingly obfuscated probability engines?
All of that is why I spend the bulk of my time testing local models and harnesses for work, rather than leaning on Gemini or Claude. It’s not that I doubt their capabilities, rather that I need to be able to show potential customers where the agent or model made a mistake that caused harm - which is something I can presently only do with local models. That’s why (I suspect) the compliance narrative from the foundational labs has been more along the lines of “humans vetting what AI does” instead of being able to prevent AI from making errors through iterative improvements on a process.
Re: The session you cannot take with you
#167Earlier quoted context omitted.
People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277 , in addition to facilitating distillation (you don't pay the full cost of RL)
The latter ("facilitating distillation") is a real concern from the labs I'm sure, but the first part I don't understand what you mean, or you misunderstand that paper, it's about "prompt injection" via manipulating the reasoning, not about the model reasoning by itself and somehow that leading to more prompt injections. They're quite literally maliciously rewriting the reasoning as the model reasons, not just showin…
Yeah, signing would prevent the paper's attack, but seeing the "raw reasoning" is helpful in figuring out what triggers refusals or (especially) what in the system prompt is guiding a specific refusal.
Gemini (the web UI) used to show raw reasoning, or at least a more detailed summary of its reasoning, less than a year ago. Complete with markdown and weird spelling idiosyncracies, so I'd lead towards "real reasoning", but who knows. The reasoning block leaked the system prompt way more often than the response block did, and you could figure out why it would refuse a request through the reasoning, even if the response itself refused to elaborate. This is, presumably, why they stopped showing it. No loss for them, just prevents "pesky users" from low hanging fruit snooping.
(Gemma 4's reasoning and output remind me strongly of what I remember Gemini 2.5/3's reasoning to be, as an aside. I guess that's obvious, but Gemma 3 felt like a totally different model, while 4 feels very Gemini-ish.)
Re: The session you cannot take with you
#168Earlier quoted context omitted.
People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277 , in addition to facilitating distillation (you don't pay the full cost of RL)
Token-based reasoning also seems like it would be inefficient, there’s no reason it has to be English or even human understandable.
What use is the reasoning if its unintelligible to a human?
Re: The session you cannot take with you
#169Earlier quoted context omitted.
People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277 , in addition to facilitating distillation (you don't pay the full cost of RL)
Or they could store the reading traces and validate the user hasn’t edited them server-side? They could sign reasoning traces so they can’t be counterfeited?
Why would the model bother to check signatures if it doesn’t check tags?
Re: The session you cannot take with you
#170Earlier quoted context omitted.
It's not even that it's hard to build a modern account system. It's that if you put your site behind a private / custom account system, and you post a link to it on HN (for example), everyone and their mother complains that it's requiring an email address and password. But if you put the same site behind a Google auth, most people wouldn't think twice to click the button.
I worked on the Google account system for a few years. I'd say it's a lot of work to beat it. Not necessarily "hard" for a team with the right skills, but certainly a lot of sweat, blood and tears. A modern account system is expected to have, in rough implementation order: email confirmations, password strength checks, password reset emails, forgot password flows (=advanced ID verification as otherwise this becomes a…
> A modern account system is expected to have, in rough implementation order: email confirmations, password strength checks, password reset emails, forgot password flows (=advanced ID verification as otherwise this becomes a backdoor into accounts), user profiles (+avatar image upload/recompression/hosting), usernames independent of email addresses along with ability to change usernames later, password brute forcing blockers, bulk signup prevention (=solid bot detection), abuse controls (can easily become a team of people), 2FA (SMS), 2FA (authenticator apps), 2FA (backup codes), 2FA (voice calls), 2FA (passkeys), 2FA: recovery when both factors are lost, enterprise SSO integration (SAML), enterprise SSO (Active Directory), fast global signout support (much harder than it looks), cookie theft mitigations, heuristic online login risk analysis to catch cases where an attacker knows the right password via phishing, support for signing the user in to multiple domains, audit logging so users can review their own sign-in history, age verification and restriction support, and possibly support for being logged in to multiple accounts in a single browser session.
You don't need half of that. Even to this day, anthropic lets you log in by sending a code to your email. No password, no dealing with resets, no MFA. So yeah, you definitely don't need all of that.