Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

191–200 of 225 posts

Re: Please do not A/B test my workflow

#191

Post by applfanboysbgon »

doc_ick wrote: post #159
applfanboysbgon wrote: post #131

Earlier quoted context omitted.

Telemetry is, ideally, collected with the intention of improving software, but that doesn't necessitate doing live A/B tests. A typical example: report hardware specs whenever the software crashes. Use that to identify some model of GPU or driver version that is incompatible with your software and figure out why. Ship a fix in the next update. What you don't do with telemetry is randomly do live experiments on your u…

Telemetry is “ideally” this. What makes you think other hosted llms (grok/xAI) don’t do this? You also got the information from asking Claude questions about its prompt, maybe it hallucinated this?

I have no idea what you're talking about or why you think I got any information from asking Claude anything. The telemetry comment was about software in general, Photoshop etc., since the person I was replying to was asking what telemetry could be for if not A/B tests. That things are injected into your prompt before sending it to their servers is trivially verified by inspecting your own outgoing packets.

Re: Please do not A/B test my workflow

#192

Post by shepherdjerred »

BAM-DevCrew wrote: post #173
chrislloyd wrote: post #166

Hi, this was my test! The plan-mode prompt has been largely unchanged since the 3.x series models and now 4.x get models are able to be successful with far less direction. My hypothesis was that shortening the plan would decrease rate-limit hits while helping people still achieve similar outcomes. I ran a few variants, with the author (and few thousand others) getting the most aggressive, limiting the plan to 40 line…

As a divergent thinker with extensive hard constraints in claude.mds and on-boarding commands that force claude to internalize my constraints, that you or some other employee of Anthropic could randomly select me for testing is horrifying. Each unexpected behavior and my corresponding reaction to it can wipe me out, my brain out, completely for hours, days, even weeks. I have in the last year spend tens (estimating a…

What?

Even without Anthropic's experimentation, anything in the context is completely probabilistic.

You cannot rely on it no matter how/how much you prompt the model

Re: Please do not A/B test my workflow

#193

Post by BiteCode_dev »

vova_hn2 wrote: post #112

Two thoughts: 1. Open source tools solve the problem of "critical functions of the application changing without notice, or being signed up for disruptive testing without opt-in". 2. This makes me afraid that it is absolutely impossible for open source tools to ever reach the level of proprietary tools like Claude Code precisely because they cannot do A/B tests like this which means that their design decisions are usu…

Let's A/B test the linux kernel, for shits and giggles.

Re: Please do not A/B test my workflow

#194

Post by shepherdjerred »

onion2k wrote: post #9

Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…

We’re still on _Hacker_ News, right?

Re: Please do not A/B test my workflow

#195

Post by Havoc »

dvfjsdhgfv wrote: post #139
Havoc wrote: post #18

Moved from CC to opencode a couple months ago because the vibes were not for me. Not bad per se but a bit too locked in and when I was looking at the raw prompts it was sending down the wire it was also quite lets call it "opinionated". Plus things like not being able to control where the websearches go. That said I have the luxury of being a hobbyist so I can accept 95% of cutting edge results for something more ope…

Can you share a setup that works for you? I found vanilla opencode vastly inferior to CC, I use it only for little toys like 3 small files that's all.

Don't think I'm doing anything particularly novel.

Using a mix of models - GLM5, MinMax 2.5 and Claude Sonnet/Opus - they find different issues

Spending fair bit of time in spec'ing things out and running all three models over it to suggest improvements / flaws & iterating till all three are happy. Same at end - look at code & suggest stability improvements. The actual writing code is GLM5 - once properly spec'd out it can generally just hammer away at it till its done

And doing a lot of microservice style architecture. Think chains of containers talking to each other over APIs

Re: Please do not A/B test my workflow

#196

Post by BAM-DevCrew »

shepherdjerred wrote: post #192
BAM-DevCrew wrote: post #173

Earlier quoted context omitted.

As a divergent thinker with extensive hard constraints in claude.mds and on-boarding commands that force claude to internalize my constraints, that you or some other employee of Anthropic could randomly select me for testing is horrifying. Each unexpected behavior and my corresponding reaction to it can wipe me out, my brain out, completely for hours, days, even weeks. I have in the last year spend tens (estimating a…

What? Even without Anthropic's experimentation, anything in the context is completely probabilistic. You cannot rely on it no matter how/how much you prompt the model

And how does one address the fragility of probabilities? Engineering. Study weaknesses and harden them. Control the probabilities. It is NOT "completely" probabilistic.

Re: Please do not A/B test my workflow

#197

Post by mrgoldenbrown »

doc_ick wrote: post #157
plussed_reader wrote: post #142

Earlier quoted context omitted.

Informed consent for a paying user is inconvenient?

Did you read the TOC?

Hiding something like this in the TOC rather than explicitly asking users to opt in is a dark pattern. You can't gain the moral highground by cackling that someone should have read the fine print.

Re: Please do not A/B test my workflow

#198

Post by oakwhiz »

chrislloyd wrote: post #166

Hi, this was my test! The plan-mode prompt has been largely unchanged since the 3.x series models and now 4.x get models are able to be successful with far less direction. My hypothesis was that shortening the plan would decrease rate-limit hits while helping people still achieve similar outcomes. I ran a few variants, with the author (and few thousand others) getting the most aggressive, limiting the plan to 40 line…

Shouldn't you be giving people their tokens back when you used their tokens to test on their environment?

Re: Please do not A/B test my workflow

#199

Post by franktankbank »

londons_explore wrote: post #176
wavefunction wrote: post #152

Earlier quoted context omitted.

A responsible company develops an informed user group they can test new changes with and receive direct feedback they can take action on.

A big tech company has ~10k experiments running at once. Some engineers will be kicking off a few experiments every day. Some will be minor things like font sizes or wording of buttons, whilst others will be entirely new features or changes in rules. Focus groups have their place, but cannot collect nearly the same scale of information.

If you have 10k experiments running then you are probably p-hacking.

Re: Please do not A/B test my workflow

#200

Post by xg15 »

krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

> The framing of A/B testing as a "silent experimentation on users"

Sorry, but how is A/B testing not exactly that? The experiments may be on non-disruptive things like button color, but they're experiments no less.

The users are also rarely informed about the experiment taking place, let alone on the motivation or evaluation criteria.

Post reply on HN