Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

201–210 of 225 posts

Re: Please do not A/B test my workflow

#201

Post by mrgoldenbrown »

onion2k wrote: post #9

Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…

So if I view source on their webpage I'm violating terms and conditions? Yikes.

Re: Please do not A/B test my workflow

#202

Post by xg15 »

krisbolton wrote: post #85
tomalbrc wrote: post #46

Earlier quoted context omitted.

Would love to know why you would consider invoking Meta “a little much”. Sounds more than appropriate.

Not to start an internet argument -- I don't think it is appropriate in this context. A/B testing the features of a web app is not unexpected or unethical. So invoking the memory of cambridge analytica (etc) is disproportionate. It's far more legitimate to just discuss how much A/B testing should negatively affect a user. I don't have an answer and it's an interesting and relevant question.

> It's far more legitimate to just discuss how much A/B testing should negatively affect a user. I don't have an answer and it's an interesting and relevant question.

You don't have an answer on "how much should A/B testing negatively affect a user"? So "a lot" would be on the table?

Re: Please do not A/B test my workflow

#203

Post by xg15 »

simianwords wrote: post #137
SlinkyOnStairs wrote: post #56

Earlier quoted context omitted.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Strange! You benefitted from all the previous a/b experiments to give you a somewhat optimal model now. But now it’s too inconvenient for you?

We actually didn't, if much of that A/B testing was to find the optimally "engagement maximizing" i.e. maximally addictive UI design.

Re: Please do not A/B test my workflow

#204

Post by londons_explore »

rkomorn wrote: post #179
londons_explore wrote: post #176

Earlier quoted context omitted.

A big tech company has ~10k experiments running at once. Some engineers will be kicking off a few experiments every day. Some will be minor things like font sizes or wording of buttons, whilst others will be entirely new features or changes in rules. Focus groups have their place, but cannot collect nearly the same scale of information.

I think a lot of people (myself included) would just like to not be constantly part of some sort of revenue optimization effort. I don't care, at all, about the "scale of information" for the company's sake.

Often the experiments are not for revenue - many of them will be optimizing user experience metrics - ie. Load time or user dropoff rate.

They are clearly good for both user satisfaction and the companies bottom line.

Re: Please do not A/B test my workflow

#205

Post by MadnessASAP »

doc_ick wrote: post #157
plussed_reader wrote: post #142

Earlier quoted context omitted.

Informed consent for a paying user is inconvenient?

Did you read the TOC?

All TOS essentially boil down to "we owe you nothing and can change the product at anytime to anything we want at our sole discretion"

Obviously it would be unreasonable to accept such terms without further context. The further context in this case being that Anthropic will maintain Claude as an AI agent and seek to improve it's performance. What is at the heart of this issue is whether or not Anthropics recent A/B testing violated that context. Not whether or not they violated the TOS (they didn't, obviously)

Re: Please do not A/B test my workflow

#206

Post by il-b »

This has been called a “vendor lock” for as long as I can remember. Congratulations, you’ve put yourself in a position where you can’t do your job without a single tool that can degrade or disappear at any moment.

Re: Please do not A/B test my workflow

#207

Post by nextzck »

chrislloyd wrote: post #166

Hi, this was my test! The plan-mode prompt has been largely unchanged since the 3.x series models and now 4.x get models are able to be successful with far less direction. My hypothesis was that shortening the plan would decrease rate-limit hits while helping people still achieve similar outcomes. I ran a few variants, with the author (and few thousand others) getting the most aggressive, limiting the plan to 40 line…

The 40-line cap not moving rate limits makes sense - plan text is cheap. The cost is in Phase 1 exploration.

Plan mode spins up to 3 explore subagents before the planner even starts, and the heuristic is "use multiple when scope is uncertain." It won't choose fewer - it's being asked to plan, so scope is always uncertain. Nothing penalizes claude for over-exploring and nothing rewards restraint.

Plan mode also ignores session state. A cold start gets the same fanout as a warm session where the relevant files are already in context. In a warm session the explore pass is pure waste - it re-reads loaded files and feeds the planner lossy summaries that conflict with what it already knows.

More tokens, worse plan.

If exploration was conditional on what's already in context..skip it for warm sessions, keep it for cold starts - that does more for both rate limits and plan quality than a hard 40-line cap.

Note: plan mode didn’t always have this 3 subagent fan out behavior attached to it, it was introduced around opus 4.6 launch.

Re: Please do not A/B test my workflow

#208

Post by jjj123 »

londons_explore wrote: post #204
rkomorn wrote: post #179

Earlier quoted context omitted.

I think a lot of people (myself included) would just like to not be constantly part of some sort of revenue optimization effort. I don't care, at all, about the "scale of information" for the company's sake.

Often the experiments are not for revenue - many of them will be optimizing user experience metrics - ie. Load time or user dropoff rate. They are clearly good for both user satisfaction and the companies bottom line.

As someone who works in these orgs, only a small fraction are about user experience metrics. 90+% are extracting more short term value with unknown second order effects on usability.

Re: Please do not A/B test my workflow

#209

Post by bartread »

chrislloyd wrote: post #166

Hi, this was my test! The plan-mode prompt has been largely unchanged since the 3.x series models and now 4.x get models are able to be successful with far less direction. My hypothesis was that shortening the plan would decrease rate-limit hits while helping people still achieve similar outcomes. I ran a few variants, with the author (and few thousand others) getting the most aggressive, limiting the plan to 40 line…

I don't mind you testing stuff out - it's the only sensible way to make the app better - but you need to give people choices to switch to different behaviours if the behaviour you're testing on them isn't working out well for them.

In other news, Claude Code login is down, so if you have time it would be sensible to proiritise fixing that:

Authorization failed Redirect URI http:/localhost:53025/callback is not supported by client.

MacOS Sequoia, VS Code 1.111.0, Firefox 147.0.4 (although also fails on Chrome 145.0.7632.160).

This just started happening as of this evening. I've tried restarting everything, and it doesn't help.

Re: Please do not A/B test my workflow

#210

Post by ryandrake »

gnabgib wrote: post #186
bcrl wrote: post #185

Earlier quoted context omitted.

Please name a computer science program that has an ethics component. Yes, I wish software developers were more like actual engineers in this regard.

All Computer Engineering & Systems Engineering programs in Canada require two ethics components (once at graduation, once at P.Eng)

Sadly, in the USA, I believe most engineering ethics classes are optional electives, and it shows when you look at the graduating student body today.
Post reply on HN