Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

Post reply on HN
Print view View original

21–30 of 225 posts

Re: Please do not A/B test my workflow

#21

Post by onion2k »

reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

The author's complaint doesn't really have anything to do with the LLM aspect of it though. They're complaining that the app silently changes what it's doing. In this case it's the injection of a prompt in a specific mode, but it could be anything really. Companies could use A/B tests on users to make Photoshop silently change the hue a user selects to be a little brighter, or Word could change the look of document titles, or a game could make enemies a bit stronger (fyi, this does actually happen - players get boosts on their first few rounds in online games to stop them being put off playing).

The complaint is about A/B tests with no visible warnings, not AI.

Re: Please do not A/B test my workflow

#22

Post by reconnecting »

danielbln wrote: post #6
reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result.

Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

Re: Please do not A/B test my workflow

#23

Post by WillAdams »

reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

Yeah, I've been using Copilot to process scans of invoices and checks (w/ a pen laid across the account information) converted to a PDF 20 at a time and it's pretty rare for it to get all 20, but it's sufficiently faster than opening them up in batches of 50 and re-saving using the Invoice ID and then using a .bat file to rename them (and remembering to quite Adobe Acrobat after each batch so that I don't run into the bug in it where it stops saving files after a couple of hundred have been so opened and re-saved).

Re: Please do not A/B test my workflow

#24

Post by freeone3000 »

danielbln wrote: post #6
reconnecting wrote: post #3

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

Yes! And it was bad then too!!

I want software that does a specific list of things, doesn’t change, and preferentially costs a known amount.

Re: Please do not A/B test my workflow

#25

Post by onion2k »

embedding-shape wrote: post #12
onion2k wrote: post #9

Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…

> To decompile, reverse engineer, disassemble, or otherwise reduce our Services to human-readable form, except when these restrictions are prohibited by applicable law. Luckily, it doesn't seem like any service was reverse-engineered or decompiled here, only a software that lived on the authors disk.

Again, read the terms. Service has a specific meaning, and it isn't what you're assuming.

Don't assume things about legal docs. You will often be wrong. Get a lawyer if it's something important.

Re: Please do not A/B test my workflow

#26

Post by applfanboysbgon »

embedding-shape wrote: post #12
onion2k wrote: post #9

Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…

> To decompile, reverse engineer, disassemble, or otherwise reduce our Services to human-readable form, except when these restrictions are prohibited by applicable law. Luckily, it doesn't seem like any service was reverse-engineered or decompiled here, only a software that lived on the authors disk.

Not "service" in human speech. Service, in bullshit legalese. They define their software as

> along with any associated apps, software, and websites (together, our “Services”)

As far as I understand, these terms actually hold up in court, too. Which is complete fucking nonsense that, I think, could only be the result of a technologically illiterate class making the decisions. Being penalised for trying to understand what software is doing on your machine is so wholly unreasonable that it should not be a valid contractual term.

Re: Please do not A/B test my workflow

#27

Post by huflungdung »

reconnecting wrote: post #22
danielbln wrote: post #6

Earlier quoted context omitted.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result . Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

[dead]

Re: Please do not A/B test my workflow

#28

Post by doc_ick »

embedding-shape wrote: post #12
onion2k wrote: post #9

Section 6.b of the Claude Code terms says they can and will change the product offering from time to time, and I imagine that means on a user segment basis rather than any implied guarantee that everyone gets the same thing. b. Subscription content, features, and services. The content, features, and other services provided as part of your Subscription, and the duration of your Subscription, will be described in the o…

> To decompile, reverse engineer, disassemble, or otherwise reduce our Services to human-readable form, except when these restrictions are prohibited by applicable law. Luckily, it doesn't seem like any service was reverse-engineered or decompiled here, only a software that lived on the authors disk.

“ I dug into the Claude Code binary.”

Re: Please do not A/B test my workflow

#29

Post by orf »

reconnecting wrote: post #22
danielbln wrote: post #6

Earlier quoted context omitted.

I don't get your point. Web tools have been doing A/B feature testing all the time, way before we had LLMs.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result . Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

It’s exactly the same as A/B testing an interface. This is just testing 4 variants of a “page” (the plan), measuring how many people pressed “continue”.
Post reply on HN