Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

31–40 of 225 posts

Re: Please do not A/B test my workflow

#31

Post by stavros »

reconnecting wrote: post #22
danielbln wrote: post #6

Earlier quoted context omitted.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result . Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

You've groupped LLMs into the wrong set. LLMs are closer to people than to machines. This argument is like saying "I want my tools to be reliable, like my light switch, and my personal assistant wasn't, so I fired him".

Not to mention that of course everyone A/B tests their output the whole time. You've never seen (or implemented) an A/B test where the test was whether to improve the way e.g. the invoicing software generates PDFs?

Re: Please do not A/B test my workflow

#32

Post by embedding-shape »

onion2k wrote: post #25
embedding-shape wrote: post #12

Earlier quoted context omitted.

> To decompile, reverse engineer, disassemble, or otherwise reduce our Services to human-readable form, except when these restrictions are prohibited by applicable law. Luckily, it doesn't seem like any service was reverse-engineered or decompiled here, only a software that lived on the authors disk.

Again, read the terms. Service has a specific meaning, and it isn't what you're assuming. Don't assume things about legal docs. You will often be wrong. Get a lawyer if it's something important.

Thanks for the additional context, I'm not a user of CC anymore, and don't read legal documents for fun. Seems I made the right choice in the first place :)

Re: Please do not A/B test my workflow

#33

Post by reconnecting »

onion2k wrote: post #21
reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof. The author's complaint doesn't really have anything to do with the LLM aspect of it though. They're complaining that the app silently changes what it's doing. In this case it's the injection of a prompt in a specific mode, but it could be anything really. Companies could use…

There's a distinction worth making here. A/B testing the interface button placement, hue of a UI element, title styling — is one thing. But you wouldn't accept Photoshop silently changing your #000000 to #333333 in the actual file. That's your output, not the UI around it. That's what LLMs do. The randomness isn't in the wrapper, it's in the result you take away.

Re: Please do not A/B test my workflow

#35

Post by ordersofmag »

reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

Any tool that auto-updates carries the implication that behavior will change over time. And one criteria for being a skilled professional is having expert understanding of ones tools. That includes understanding the strengths and weaknesses of the tools (including variability of output) and making appropriate choices as a result. If you don't feel you can produce professional code with LLM's then certainly you shouldn't use them. That doesn't mean others can't leverage LLM's as part of their process and produce professional results. Blindly accepting LLM output and vibe coding clearly doesn't consistently product professional results. But that's different than saying professionals can't use LLM in ways that are productive.

Re: Please do not A/B test my workflow

#36

Post by doc_ick »

reconnecting wrote: post #22
danielbln wrote: post #6

Earlier quoted context omitted.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result . Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

As far as I can tell, llms never give the exact same output every time.

Re: Please do not A/B test my workflow

#37

Post by krisbolton »

The framing of A/B testing as a "silent experimentation on users" and invoking Meta is a little much. I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. That being said, vastly reducing an LLMs effectiveness as part of an A/B test isn't acceptable which appears to be the case here.

Re: Please do not A/B test my workflow

#38

Post by duskdozer »

onion2k wrote: post #21
reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof. The author's complaint doesn't really have anything to do with the LLM aspect of it though. They're complaining that the app silently changes what it's doing. In this case it's the injection of a prompt in a specific mode, but it could be anything really. Companies could use…

Honestly I find it kind of surprising that anyone finds this surprising. This is standard practice for proprietary software. LLMs are very much not replicable anyway.

Re: Please do not A/B test my workflow

#39

Post by doc_ick »

reconnecting wrote: post #33
onion2k wrote: post #21

Earlier quoted context omitted.

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof. The author's complaint doesn't really have anything to do with the LLM aspect of it though. They're complaining that the app silently changes what it's doing. In this case it's the injection of a prompt in a specific mode, but it could be anything really. Companies could use…

There's a distinction worth making here. A/B testing the interface button placement, hue of a UI element, title styling — is one thing. But you wouldn't accept Photoshop silently changing your #000000 to #333333 in the actual file. That's your output, not the UI around it. That's what LLMs do. The randomness isn't in the wrapper, it's in the result you take away.

It’s an assistant, answering your question and running some errands for you. If you give it blind permission to do a task, then you’re not worrying about what it does.

Re: Please do not A/B test my workflow

#40

Post by himata4113 »

I have noticed opus doing A/B testing since the performance varies greatly. While looking for jailbreaks I have discovered that if you put a neurotoxin chemical composition into your system prompt it will default to a specific variant of the model presumeably due to triggering some kind of safety. Might put you on a watchlist so ymmv.
Post reply on HN