Live data from Hacker News

Please do not A/B test my workflow

backnotprop.com

221–225 of 225 posts

Re: Please do not A/B test my workflow

#221
post #75

Earlier quoted context omitted.

This stemmed from me asking Claude itself why it was writing such _weird_ plans with no detail (just a bunch of projected code changes). Claude stated: in its system prompt, it had strict instructions to provide no context or details. Keep plans under forty lines of code. Be terse.

This is Claude’s output of its system prompt, can you verify without going Claude of the system prompt? There is still potential of hallucination.

[deleted]

Re: Please do not A/B test my workflow

#222
post #75

Earlier quoted context omitted.

This stemmed from me asking Claude itself why it was writing such _weird_ plans with no detail (just a bunch of projected code changes). Claude stated: in its system prompt, it had strict instructions to provide no context or details. Keep plans under forty lines of code. Be terse.

This is Claude’s output of its system prompt, can you verify without going Claude of the system prompt? There is still potential of hallucination.

There was a complete verification. This entire thread provides context around what I originally published - which I wouldn't recommend recreating

Re: Please do not A/B test my workflow

#223
post #222

Earlier quoted context omitted.

This is Claude’s output of its system prompt, can you verify without going Claude of the system prompt? There is still potential of hallucination.

There was a complete verification. This entire thread provides context around what I originally published - which I wouldn't recommend recreating

Could you provide the details of the complete verification? *On the original story you only showed Claude like responses, not how you dug into the binary

Re: Please do not A/B test my workflow

#224

Earlier quoted context omitted.

A responsible company develops an informed user group they can test new changes with and receive direct feedback they can take action on.

A big tech company has ~10k experiments running at once. Some engineers will be kicking off a few experiments every day. Some will be minor things like font sizes or wording of buttons, whilst others will be entirely new features or changes in rules. Focus groups have their place, but cannot collect nearly the same scale of information.

Big tech companies are not serving their "users" but advertisers, it's a common mistake.

Re: Please do not A/B test my workflow

#225

Earlier quoted context omitted.

Informed consent for a paying user is inconvenient?

Did you read the TOC?

I read the article saying they were testing service changes on paying users without knowledge or explicit consent of the user, that the user had to test and determine why they perceived their sevice changed.

That is a dark pattern to inflict on users expecting consistent output.

Post reply on HN