Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

51–60 of 225 posts

Re: Please do not A/B test my workflow

#51

Post by risyachka »

EMM_386 wrote: post #47
rusakov-field wrote: post #43

On one side I am frustrated with LLMs because they derail you by throwing grammatically correct bullshit and hallucinations at you, where if you slip and entertain some of it momentarily it might slow you down. But on the other hand they are so useful with boilerplate and connecting you with verbiage quickly that might guide you to the correct path quicker than conventional means. Like a clueless CEO type just spitba…

> But you REALLY need to know your stuff to begin with for they to be of any use. Those who think they will take over are clueless. Or - there are enough people who know their stuff that the people who don't will be replaced and they will take over anyway.

> there are enough people who know their stuff

unless the bar for "know their stuff" is very very low - this is not the case in the nearest future

Re: Please do not A/B test my workflow

#52

Post by Mc_Big_G »

rusakov-field wrote: post #43

On one side I am frustrated with LLMs because they derail you by throwing grammatically correct bullshit and hallucinations at you, where if you slip and entertain some of it momentarily it might slow you down. But on the other hand they are so useful with boilerplate and connecting you with verbiage quickly that might guide you to the correct path quicker than conventional means. Like a clueless CEO type just spitba…

>Those who think they will take over are clueless.

You're underestimating where it's headed.

Re: Please do not A/B test my workflow

#53

Post by johnisgood »

reconnecting wrote: post #22
danielbln wrote: post #6

Earlier quoted context omitted.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result . Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

> same input, different result.

What is your point? You get this from LLMs. It does not mean that it is not useful.

Re: Please do not A/B test my workflow

#54

Post by lwhi »

gruez wrote: post #20
nemo44x wrote: post #13

They lose money at $200/month in most cases. Again, the old rules still apply. You are the product.

>They lose money at $200/month in most cases. Source? Every time I see claims on profitability it's always hand wavy justifications.

'Hand wavy' is one of my LLMs favourite terms.

Re: Please do not A/B test my workflow

#56

Post by SlinkyOnStairs »

krisbolton wrote: post #37

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo.

I disagree in the case of LLMs.

AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output".

It's already enough of a pain in the ass to constrain these systems without the companies silently changing things around.

And this also pretty much ruins any attempt to research Claude Code's long term effectiveness in an organisation. Any negative result can now be thrown straight into the trash because of the chance Anthropic put you on the wrong side of an A/B test.

> That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

The open question here is whether or not they were doing similar things to their other products. Claude Code shitting out a bad function is annoying but should be caught in review.

People use LLMs for things like hiring. An undeclared A-B test there would be ethically horrendous and a legal nightmare for the client.

Re: Please do not A/B test my workflow

#57

Post by Mtinie »

reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

What would you do differently if LLM outputs were deterministic?

Perhaps I approach this from a different perspective than you do, so I’m interested to understand other viewpoints.

I review everything that my models produce the same way I review work from my coworkers: Trust but verify.

Re: Please do not A/B test my workflow

#58

Post by applfanboysbgon »

duskdozer wrote: post #38
onion2k wrote: post #21

Earlier quoted context omitted.

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof. The author's complaint doesn't really have anything to do with the LLM aspect of it though. They're complaining that the app silently changes what it's doing. In this case it's the injection of a prompt in a specific mode, but it could be anything really. Companies could use…

Honestly I find it kind of surprising that anyone finds this surprising. This is standard practice for proprietary software. LLMs are very much not replicable anyway.

This is in no way standard practice for proprietary software, WTF is with you dystopian weirdos trying to gaslight people? Adobe's suite incl. Photoshop does not do this, Microsoft Office incl. Excel does not do this, professional video editing software does not do this, professional music production software does not do this, game engines do not do this. That short list probably covers 80-90% of professional software usage alone. People do this when serving two versions of a website, but doing this on software that runs on my machine is frankly completely unacceptable and in no way normal.

Re: Please do not A/B test my workflow

#59

Post by bushido »

I have no issues with A/B tests.

I do have an issue with the plan mode. And nine out of ten times, it is objectively terrible. The only benefit I've seen in the past from using plan mode is it remembers more information between compactions as compared to the vanilla - non-agent team workflow.

Interestingly, though, if you ask it to maintain a running document of what you're discussing in a markdown file and make it create an evergreen task at the top of its todo list which references the markdown file and instructs itself to read it on every compaction, you get much better results.

Re: Please do not A/B test my workflow

#60

Post by johnisgood »

applfanboysbgon wrote: post #49
stavros wrote: post #31

Earlier quoted context omitted.

You've groupped LLMs into the wrong set. LLMs are closer to people than to machines. This argument is like saying "I want my tools to be reliable, like my light switch, and my personal assistant wasn't, so I fired him". Not to mention that of course everyone A/B tests their output the whole time. You've never seen (or implemented) an A/B test where the test was whether to improve the way e.g. the invoicing software g…

> LLMs are closer to people than to machines. jfc. I don't have anything to say to this other than that it deserves calling out. > You've never seen (or implemented) an A/B test where the test was whether to improve the way e.g. the invoicing software generates PDFs? I have never in my life seen or implemented an a/b test on a tool used by professionals. I see consumer-facing tests on websites all the time, but nothi…

Why are you calling it out? You are interpreting the statement too literally. The point is probably about behavior, not nature. LLMs do not always produce identical outputs for identical prompts, which already makes them less like deterministic machines and superficially closer to humans in interaction. That is it. The comparison can end here.
Post reply on HN