Please do not A/B test my workflow • Hacker News Board

Re: Please do not A/B test my workflow

#81

Post by andrewaylett » Sat, Mar 14, 2026, 12:57 PM UTC

bushido wrote: post #59

I have no issues with A/B tests. I do have an issue with the plan mode. And nine out of ten times, it is objectively terrible. The only benefit I've seen in the past from using plan mode is it remembers more information between compactions as compared to the vanilla - non-agent team workflow. Interestingly, though, if you ask it to maintain a running document of what you're discussing in a markdown file and make it c…

> on every compaction

I've only hit the compaction limit a handful of times, and my experience degraded enough that I work quite hard to not hit it again.

One thing I like about the current implementation of plan mode is that it'll clear context -- so if I complete a plan, I can use that context to write the next plan without growing context without bound.

Re: Please do not A/B test my workflow

#82

Post by pshirshov » Sat, Mar 14, 2026, 12:57 PM UTC

sunaookami wrote: post #76

pshirshov wrote: post #70

> I pay $200/month for Claude Code Which is still very cheap. There are other options, local Qwen 3.5 35b + claude code cli is, in my opinion, comparable in quality with Sonnet 4..4.5 - and without a/b tests!

In what world is 200$ per month cheap?

Where the value you extract out of the model is orders of magnitude higher than the price of 2..6 hours of your time.

Re: Please do not A/B test my workflow

#83

Post by Kiro » Sat, Mar 14, 2026, 12:58 PM UTC

sunaookami wrote: post #76

pshirshov wrote: post #70

> I pay $200/month for Claude Code Which is still very cheap. There are other options, local Qwen 3.5 35b + claude code cli is, in my opinion, comparable in quality with Sonnet 4..4.5 - and without a/b tests!

In what world is 200$ per month cheap?

It's not cheap but it's also not unusual for devs to burn $200 a day on tokens.

Re: Please do not A/B test my workflow

#84

Post by johnisgood » Sat, Mar 14, 2026, 12:59 PM UTC

Apparently the blog stripped the decompilation details for ToS reasons, which sucks because those are exactly the hack-y bits that make this interesting for HN.

> It told me it was following specific system instructions to hard-cap plans at 40 lines, forbid context sections, and “delete prose, not file paths.

Yeah, would be nice to be able to view and modify these instructions.

Re: Please do not A/B test my workflow

#85

Post by krisbolton » Sat, Mar 14, 2026, 12:59 PM UTC

tomalbrc wrote: post #46

krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

Would love to know why you would consider invoking Meta “a little much”. Sounds more than appropriate.

Not to start an internet argument -- I don't think it is appropriate in this context. A/B testing the features of a web app is not unexpected or unethical. So invoking the memory of cambridge analytica (etc) is disproportionate. It's far more legitimate to just discuss how much A/B testing should negatively affect a user. I don't have an answer and it's an interesting and relevant question.

Re: Please do not A/B test my workflow

#86

Post by raw_anon_1111 » Sat, Mar 14, 2026, 1:00 PM UTC

SlinkyOnStairs wrote: post #56

krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Would you rather they change things for everyone at once without testing?

Re: Please do not A/B test my workflow

#87

Post by mikkupikku » Sat, Mar 14, 2026, 1:00 PM UTC

applfanboysbgon wrote: post #71

mikkupikku wrote: post #69

Earlier quoted context omitted.

What other tool can I have a conversation with? I can't talk to a keyboard as if it were a coworker. Consider this seriously, instead of just letting your gut reaction win. Coding with claude code is much closer to pair programming than it is to anything else.

You could have a conversation with Eliza, SmarterChild, Siri, or Alexa. I would say surely you don't consider Eliza to be closer to person than machine, but then it takes a deeply irrational person to have led to this conversation in the first place so maybe you do.

Not productive conversations. If you had ever made a serious attempt to use these technologies instead of trying to come up with excuses to ignore it, you would not even think of comparing a modern LLM coding agent to some gimmick like Alexa or ELIZA. Seriously, get real.

Re: Please do not A/B test my workflow

#88

Post by everdrive » Sat, Mar 14, 2026, 1:00 PM UTC

krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

>I don't believe A/B testing is an inherent evil,

Evil might be a stretch, but I really hate A/B testing. Some feature or UI component you relied on is now different, with no warning, and you ask a coworker about it, and they have no idea what you're talking about.

Usually, the change is for the worse, but gets implemented anyway. I'm sure the teams responsible have "objective" "data" which "proves" it's the right direction, but the reality of it is often the opposite.

Re: Please do not A/B test my workflow

#89

Post by casey2 » Sat, Mar 14, 2026, 1:01 PM UTC

This blog looks like an ad for Claude, all it's posts are about Claude and it was made in 2026

Re: Please do not A/B test my workflow

#90

Post by applfanboysbgon » Sat, Mar 14, 2026, 1:03 PM UTC

mikkupikku wrote: post #87

applfanboysbgon wrote: post #71

Earlier quoted context omitted.

You could have a conversation with Eliza, SmarterChild, Siri, or Alexa. I would say surely you don't consider Eliza to be closer to person than machine, but then it takes a deeply irrational person to have led to this conversation in the first place so maybe you do.

Not productive conversations. If you had ever made a serious attempt to use these technologies instead of trying to come up with excuses to ignore it, you would not even think of comparing a modern LLM coding agent to some gimmick like Alexa or ELIZA. Seriously, get real.

Not only have I used the technology, I've worked for a startup that serves its own models. When you work with the technology, it could not be more obvious that you are programming software, and that there is nothing even remotely person-like about LLMs. To the extent that people think so, it is sheer ignorance of the basic technicals, in exactly the same way that ELIZA fooled non-programmers in the 1960s. You'd think we'd have collectively learned something in the 60 years since but I suppose not.