Seems like a straightforward solution would be to get people to opt-in by offering them credits, increased limits, early access to new features, etc.
Universities have IRBs for good reasons.
Please do not A/B test my workflow
121–130 of 225 posts
Re: Please do not A/B test my workflow
#122Earlier quoted context omitted.
> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…
Long term effectiveness? LLMs are such a fast moving target. Suppose anthropic reached out to you and gave you a model id you could pin down for the next year to freeze any a/b tests. Would you really want that? Next month a new model could be released to everyone else - or by a competitor - that’s a big step difference in performance in tasks you care about. You’d rather be on your own path learning about the state…
>Suppose anthropic reached out to you and gave you a model id you could pin down for the next year to freeze any a/b tests. Would you really want that?
Where can I sign up?
Re: Please do not A/B test my workflow
#123Here’s the original article which was much more informative and interesting: https://web.archive.org/web/20260314105751/https://backnotpr... Can’t believe HN has become so afraid of generic probably-unenforceable “plz don’t reverse engineer” EULAs. We deserve to know what these tools are doing. I’ve seen poor results from plan mode recently too and this explains a lot.
> probably-unenforceable
It's very easy to just ban the user and if your whole workflow relies on the tool, you really don't want it.
Re: Please do not A/B test my workflow
#124Earlier quoted context omitted.
> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…
I think you would be hard pushed to find any big tech company which doesn't do some kind of A B testing. It's pretty much required if you want to build a great product.
A/B testing is the child of profit maximization, engagement farming, and enshittification. Not of "great product building".
Re: Please do not A/B test my workflow
#125Seems like a straightforward solution would be to get people to opt-in by offering them credits, increased limits, early access to new features, etc. Universities have IRBs for good reasons.
A problem with this approach could be that you're now only testing the feature with the kind of people who would sign up for an A/B test. This group may not be representative of your whole user-base.
Re: Please do not A/B test my workflow
#126Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…
When a company tells you not to reverse, decompile, or disassemble their software, the first thing you should do is just that.
Re: Please do not A/B test my workflow
#127Earlier quoted context omitted.
Why are you calling it out? You are interpreting the statement too literally. The point is probably about behavior, not nature. LLMs do not always produce identical outputs for identical prompts, which already makes them less like deterministic machines and superficially closer to humans in interaction. That is it. The comparison can end here.
They actually can, though. The frontier model providers don't expose seeds, but for inferencing LLMs on your own hardware, you can set a specific seed for deterministic output and evaluate how small changes to the context change the output on that seed. This is like suggesting that Photoshop would be "more like a person than a machine" if they added a random factor every time you picked a color that changed the value…
You are right, I was wrong. I think anthropomorphizing LLMs to begin with is kind of silly. The whole "LLMs are closer to people than to machines" comparison is misleading, especially when the argument comes down to output variability.
Their outputs can vary in ways that superficially resemble human variability, but variability alone is a poor analogy for humanness. A more meaningful way to compare is to look at functional behaviors such as "pattern recognition", "contextual adaptation", "generalization to new prompts", and "multi-step reasoning". These behaviors resemble aspects of human capabilities. In particular, generalization allows LLMs to produce coherent outputs for tasks they were not explicitly trained on, rather than just repeating training data, making it a more meaningful measure than randomness alone.
That said, none of this means LLMs are conscious, intentional, or actually understanding anything. I am glad you brought up the seed and determinism point. People should know that you can make outputs fully predictable, so the "human-like" label mostly only shows up under stochastic sampling. It is far more informative to look at real functional capabilities instead of just variability, and I think more people should be aware of this.
Re: Please do not A/B test my workflow
#128Two thoughts: 1. Open source tools solve the problem of "critical functions of the application changing without notice, or being signed up for disruptive testing without opt-in". 2. This makes me afraid that it is absolutely impossible for open source tools to ever reach the level of proprietary tools like Claude Code precisely because they cannot do A/B tests like this which means that their design decisions are usu…
Regarding point 1 specifically, there were so many people seriously miffed at the “man after midnight”[0] time-based easter egg that I would be careful with that reasoning.
Open source doesn’t always mean reproducible.
People don’t enjoy the thought of auditing code… someone else will do it; and its made somewhat worse with our penchant to pull in half the universe as dependencies (Rust, Go and Javascript tend to lean in this direction to various extremes). But auditing would be necessary in order for your first point here to be as valid as you present.
[0]: https://gitlab.com/man-db/man-db/-/commit/002a6339b1fe8f83f4...
Re: Please do not A/B test my workflow
#129The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.
>I don't believe A/B testing is an inherent evil, Evil might be a stretch, but I really hate A/B testing. Some feature or UI component you relied on is now different, with no warning, and you ask a coworker about it, and they have no idea what you're talking about. Usually, the change is for the worse, but gets implemented anyway. I'm sure the teams responsible have "objective" "data" which "proves" it's the right di…
> I'm sure the teams responsible have "objective" "data" which "proves" it's the right direction, but the reality of it is often the opposite.
In my experience all manner of analytics data frequently gets misused to support whatever narrative the product manager wants it to support.
With enough massaging you can make “objective” numbers say anything, especially if you do underhanded things like bury a previously popular feature three modals deep or put it behind a flag. “Oh would you look at that, nobody uses this feature any more! Must be safe to remove it.”
Re: Please do not A/B test my workflow
#130It seems a bit odd to complain "I need transparency into how it works and the ability to configure it" when his workflow is already relying on a black box with zero transparency into how it works.