Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

111–120 of 225 posts

Re: Please do not A/B test my workflow

#111

Post by embedding-shape »

londons_explore wrote: post #99
SlinkyOnStairs wrote: post #56

Earlier quoted context omitted.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

I think you would be hard pushed to find any big tech company which doesn't do some kind of A B testing. It's pretty much required if you want to build a great product.

Yeah, that's why we didn't have anything anyone could possibly consider as a "great product" until A/B testing existed as a methodology.

Or, you could, you know, try to understand your users without experimenting on them, like countless of others have managed to do before, and still shipped "great products".

Re: Please do not A/B test my workflow

#112

Post by vova_hn2 »

Two thoughts:

1. Open source tools solve the problem of "critical functions of the application changing without notice, or being signed up for disruptive testing without opt-in".

2. This makes me afraid that it is absolutely impossible for open source tools to ever reach the level of proprietary tools like Claude Code precisely because they cannot do A/B tests like this which means that their design decisions are usually informed by intuition and personal experience but not by hard data collected at scale.

Re: Please do not A/B test my workflow

#113

Post by some_random »

doc_ick wrote: post #39
reconnecting wrote: post #33

Earlier quoted context omitted.

There's a distinction worth making here. A/B testing the interface button placement, hue of a UI element, title styling — is one thing. But you wouldn't accept Photoshop silently changing your #000000 to #333333 in the actual file. That's your output, not the UI around it. That's what LLMs do. The randomness isn't in the wrapper, it's in the result you take away.

It’s an assistant, answering your question and running some errands for you. If you give it blind permission to do a task, then you’re not worrying about what it does.

That's not what they're doing, they are trying to use plan mode to plan out a task. I don't know where you could have got the idea that they were blindly doing anything.

Re: Please do not A/B test my workflow

#114

Post by duskdozer »

ozgrakkurt wrote: post #19
onion2k wrote: post #9

Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…

Why should anyone care about their TOS while they are laundering people’s work at a massive scale?

Because by contrast they have the money and institutional capture to make your life miserable if you don't.

Re: Please do not A/B test my workflow

#115

Post by vova_hn2 »

airza wrote: post #106
SlinkyOnStairs wrote: post #56

Earlier quoted context omitted.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Isn’t the horrendous ethical and legal decision delegating your hiring process to a black box?

> ethical and legal decision

These are two very different things. I suspect that in some cases pointing finger at a black box instead of actually explaining your decisions can actually shield you from legal liability...

Re: Please do not A/B test my workflow

#116

Post by aeinbu »

raw_anon_1111 wrote: post #86
SlinkyOnStairs wrote: post #56

Earlier quoted context omitted.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Would you rather they change things for everyone at once without testing?

That is not the only other alternative.

You can do A/B testing splitting up your audience in groups, having some audience use A, and others use B - all the time.

I think the article’s author is frustrated over sometimes getting A and at other times B, and not knowing when he is on either.

Re: Please do not A/B test my workflow

#117

Post by paulryanrogers »

vova_hn2 wrote: post #115
airza wrote: post #106

Earlier quoted context omitted.

Isn’t the horrendous ethical and legal decision delegating your hiring process to a black box?

> ethical and legal decision These are two very different things. I suspect that in some cases pointing finger at a black box instead of actually explaining your decisions can actually shield you from legal liability...

For some proponents, AI is liability washing

Re: Please do not A/B test my workflow

#118

Post by bartread »

There’s more than a bit of irony in the author complaining about A/B testing and then, because they’re getting a lot of traffic and attention on HN, removing key content that was originally in their piece so some of us have seen it but many of us won’t.

Whilst I broadly agree with their point, colour me unimpressed by this behaviour.

EDIT: God bless archive.org: https://web.archive.org/web/20260314105751/https://backnotpr.... This provides a lot more useful insight that, to me, significantly strengthens the point the article is making. Doesn’t mean I’m going to start picking apart binaries (though it wouldn’t be the first time), but how else are you supposed to really understand - and prove - what’s going on unless you do what the author did? Point is, it’s a much better, more useful, and more interesting article in its uncensored form.

EDIT 2: For me it’s not the fact that Anthropic are doing these tests that’s the problem: it’s that they’re not telling us, and they’re not giving us a way to select a different behaviour (which, if they did, would also give them useful insights into users needs).

Re: Please do not A/B test my workflow

#119

Post by duskdozer »

applfanboysbgon wrote: post #58
duskdozer wrote: post #38

Earlier quoted context omitted.

Honestly I find it kind of surprising that anyone finds this surprising. This is standard practice for proprietary software. LLMs are very much not replicable anyway.

This is in no way standard practice for proprietary software, WTF is with you dystopian weirdos trying to gaslight people? Adobe's suite incl. Photoshop does not do this, Microsoft Office incl. Excel does not do this, professional video editing software does not do this, professional music production software does not do this, game engines do not do this. That short list probably covers 80-90% of professional softwar…

Maybe then, it's just my expectation of what they would be doing. What else is all the telemetry for? As a side note, my impression is that this is less of a photoshop and more of a website situation in that most of the functionality is input and response to/from their servers.

Re: Please do not A/B test my workflow

#120

Post by [deleted] »

applfanboysbgon wrote: post #96
mikkupikku wrote: post #94

Earlier quoted context omitted.

I really don't care where you've worked, to seriously argue that LLMs aren't more capable of conversation than ELIZA, aren't capable of pair programming even, is gargantuan levels of cope.

I didn't make any claims about their utility. I said that they are not like people. They are machines through and through. Regular software programs. Programs that are, I suppose, a little bit too complex for the average human to understand, so now we have the Eliza effect applying to an entirely new generation. "I had not realized ... exposures to a relatively simple computer program could induce powerful delusional…

[deleted]
Post reply on HN