Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

101–110 of 225 posts

Re: Please do not A/B test my workflow

#101

Post by samdjstephens »

andrewaylett wrote: post #81
bushido wrote: post #59

I have no issues with A/B tests. I do have an issue with the plan mode. And nine out of ten times, it is objectively terrible. The only benefit I've seen in the past from using plan mode is it remembers more information between compactions as compared to the vanilla - non-agent team workflow. Interestingly, though, if you ask it to maintain a running document of what you're discussing in a markdown file and make it c…

> on every compaction I've only hit the compaction limit a handful of times, and my experience degraded enough that I work quite hard to not hit it again. One thing I like about the current implementation of plan mode is that it'll clear context -- so if I complete a plan, I can use that context to write the next plan without growing context without bound.

I really like this too - having the previous plan and implementation in place to create the next plan, but then clearing context once that next plan exists feels like a great way to have exactly the right context at the right time.

I often do follow ups, that would have been short message replies before, as plans, just so I can clear context once it’s ready. I’m hitting the context limit much less often now too.

Re: Please do not A/B test my workflow

#102

Post by steve-atx-7600 »

SlinkyOnStairs wrote: post #56
krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Long term effectiveness? LLMs are such a fast moving target. Suppose anthropic reached out to you and gave you a model id you could pin down for the next year to freeze any a/b tests. Would you really want that? Next month a new model could be released to everyone else - or by a competitor - that’s a big step difference in performance in tasks you care about. You’d rather be on your own path learning about the state of the world that doesn’t exist anymore? nov-ish 2025 and after, for example, seemed like software engineering changed forever because of improvements in opus.

Re: Please do not A/B test my workflow

#103

Post by qazxcvbnmlp »

rusakov-field wrote: post #43

On one side I am frustrated with LLMs because they derail you by throwing grammatically correct bullshit and hallucinations at you, where if you slip and entertain some of it momentarily it might slow you down. But on the other hand they are so useful with boilerplate and connecting you with verbiage quickly that might guide you to the correct path quicker than conventional means. Like a clueless CEO type just spitba…

One of the main skills of using the llm well is knowing the difference between useful output and ai slop.

Re: Please do not A/B test my workflow

#104

Post by r_lee »

dkersten wrote: post #14
reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

Anthropic have done a lot of things that would give me pause about trusting them in a professional context. They are anything but transparent, for example about the quota limits. Their vibe coded Claude code cli releases are a buggy mess too. Also the model quality inconsistency: before a new model release, there’s a week or two where their previous model is garbage. A/B testing is fine in itself, you need to learn a…

> vibe coded Claude code cli releases are a buggy mess too

this is what gets me.

are they out of money? are so desperate to penny pinch that they can't just do it properly?

what's going on in this industry?

Re: Please do not A/B test my workflow

#105

Post by steve-atx-7600 »

steve-atx-7600 wrote: post #102
SlinkyOnStairs wrote: post #56

Earlier quoted context omitted.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Long term effectiveness? LLMs are such a fast moving target. Suppose anthropic reached out to you and gave you a model id you could pin down for the next year to freeze any a/b tests. Would you really want that? Next month a new model could be released to everyone else - or by a competitor - that’s a big step difference in performance in tasks you care about. You’d rather be on your own path learning about the state…

If you really want to keep non-determinism down, you could try (1) see if you can fix the installed version of the clause code client app (I haven’t looked into the details to prevent auto-updating..because bleeding edge person) and (2) you can pin to a specific model version which you think would have to reduce a/b test exposure to some extent https://support.claude.com/en/articles/11940350-claude-code-...

Edit: how to disable auto updates of the client app https://code.claude.com/docs/en/setup#disable-auto-updates

Re: Please do not A/B test my workflow

#106

Post by airza »

SlinkyOnStairs wrote: post #56
krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…

Isn’t the horrendous ethical and legal decision delegating your hiring process to a black box?

Re: Please do not A/B test my workflow

#107

Post by hollow-moe »

krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

Tech companies really have issues with "informed and conscious consent" doesn't they

Re: Please do not A/B test my workflow

#108

Post by [deleted] »

applfanboysbgon wrote: post #64
johnisgood wrote: post #60

Earlier quoted context omitted.

Why are you calling it out? You are interpreting the statement too literally. The point is probably about behavior, not nature. LLMs do not always produce identical outputs for identical prompts, which already makes them less like deterministic machines and superficially closer to humans in interaction. That is it. The comparison can end here.

They actually can, though. The frontier model providers don't expose seeds, but for inferencing LLMs on your own hardware, you can set a specific seed for deterministic output and evaluate how small changes to the context change the output on that seed. This is like suggesting that Photoshop would be "more like a person than a machine" if they added a random factor every time you picked a color that changed the value…

[deleted]

Re: Please do not A/B test my workflow

#109

Post by pinum »

Here’s the original article which was much more informative and interesting:

https://web.archive.org/web/20260314105751/https://backnotpr...

Can’t believe HN has become so afraid of generic probably-unenforceable “plz don’t reverse engineer” EULAs. We deserve to know what these tools are doing.

I’ve seen poor results from plan mode recently too and this explains a lot.

Re: Please do not A/B test my workflow

#110

Post by gruez »

nemo44x wrote: post #100
gruez wrote: post #20

Earlier quoted context omitted.

>They lose money at $200/month in most cases. Source? Every time I see claims on profitability it's always hand wavy justifications.

There’s a lot of articles about it. It costs them $500+ for heavy users. They do this to capture market share and also to train their agent loops with human reinforcement learning. https://ezzekielnjuguna.medium.com/why-anthropic-is-practica...

>There’s a lot of articles about it. ....

>https://ezzekielnjuguna.medium.com/why-anthropic-is-practica...

You chose a bad one. It just asserts the 95% figure without evidence and then uses it as the premise for the rest of the article. That just confirms what I said earlier about how "Every time I see claims on profitability it's always hand wavy justifications.". Moreover the article reeks of LLM-isms.

Post reply on HN