For those confused about this submission: the original post is here:
https://web.archive.org/web/20260314105751/https://backnotpr...
Please do not A/B test my workflow
141–150 of 225 posts
Re: Please do not A/B test my workflow
#142Earlier quoted context omitted.
> I don't believe A/B testing is an inherent evil, you need to get the test design right, and that would be better framing for the post imo. I disagree in the case of LLMs. AI already has a massive problem in reproducibility and reliability, and AI firms gleefully kick this problem down to the users. "Never trust it's output". It's already enough of a pain in the ass to constrain these systems without the companies s…
Strange! You benefitted from all the previous a/b experiments to give you a somewhat optimal model now. But now it’s too inconvenient for you?
Informed consent for a paying user is inconvenient?
Re: Please do not A/B test my workflow
#143Earlier quoted context omitted.
Regarding point 1 specifically, there were so many people seriously miffed at the “man after midnight ”[0] time-based easter egg that I would be careful with that reasoning. Open source doesn’t always mean reproducible. People don’t enjoy the thought of auditing code… someone else will do it; and its made somewhat worse with our penchant to pull in half the universe as dependencies (Rust, Go and Javascript tend to le…
> People don’t enjoy the thought of auditing code… someone else will do it I think that with modern LLMs auditing a big project personally, instead of relying on someone else to do it, actually became more realistic. You can ask an LLM to walk you through the code, highlight parts that seem unusual or suspicious, etc. On the other hand, LLMs also made producing code cheaper then ever, so you can argue, that big proje…
fuck man, I'm either seriously stupid or y'all are taking crazy pills.
LLMs are auto-complete on steroids; I've lived through enough iterations of Markov Chains giving semi-sensible output (that we give meaning to) and neural networks which present the illusion of intelligence to see directly what these LLMs are: a fuckload of compute designed to find "the next most common word" given the preceding 10,000 or more words.
In such a case, the idea of it actually auditing anything is hilarious. You're looking at a 1/100 in actually finding anything useful. It will find "issues" in things that aren't issues (because they are covered by other cases), or skip over issues that people have historically had hard time identifying themselves.
It's not running code in a sandbox and watching memory, it's not making logical maps of code paths in its mind, it's not reasoning at all. It's fucking autocomplete. Stop treating it as if it can think, it fucking can't.
I'm so tired of this hype. It's very easy to convince midwits that something is intelligent, I'm absolutely not surprised at how salesmen and con-men operate now that I've seen this first hand.
Re: Please do not A/B test my workflow
#144It seems a bit odd to complain "I need transparency into how it works and the ability to configure it" when his workflow is already relying on a black box with zero transparency into how it works.
There's a difference between "LLMs are inherently black boxes that require lots of work to attempt to understand" and explicitly changing how a piece of software works. Should people not complain about unannounced changes to the contents of their food or medicine because we don't understand everything about how the human body works?
Except the system prompt that gets prepended to your own prompt is part of the black box, and obviously should be expected to change over time. You are also told that you're not allowed to reverse engineer it. Even in the absence of the system prompt being changed, the output of the LLM is non-deterministic.
I'm not sure I understand your last analogy. How would changes to the human body change the contents of the food that is eaten? It would be more analogous to compare it with unexpected changes to the body's output given the same inputs as previously, a phenomenon humans frequently experience.
Re: Please do not A/B test my workflow
#145For anyone else wondering why the article ends in a non-sequitur: it looks like the author wrote about decompiling the Claude Code binaries and (presumably) discovering A/B testing paths in the code. HN user 'onion2k pointed out that doing this breaks Anthropic's T&Cs: https://news.ycombinator.com/item?id=47375787
[deleted]
Re: Please do not A/B test my workflow
#146Two thoughts: 1. Open source tools solve the problem of "critical functions of the application changing without notice, or being signed up for disruptive testing without opt-in". 2. This makes me afraid that it is absolutely impossible for open source tools to ever reach the level of proprietary tools like Claude Code precisely because they cannot do A/B tests like this which means that their design decisions are usu…
A/B test doesn't necessarily imply improvements for the user. It could be testing of future enshittification methods. See YouTube for an example.
Re: Please do not A/B test my workflow
#147Earlier quoted context omitted.
The last time I did contract work when I was between jobs I made $100/hour. And I won’t say how much my employer charges for me. But you can see how much the major consulting companies charge here https://ceriusexecutives.com/management-consultants-whats-th...
Claude will happily generate tons of useless code and you will be charged appropriately. the output of LLMs has nothing to do with payment rates, otherwise you end up with absurdities like valuating useless CCC that was very expensive to build using LOCs as a metrics whereas in reality is a toy product nobody in their right mind would ever use.
My metrics are really simple - I don’t do staff augmentation. I get a contract (SOW) with a known set of requirements and acceptance criteria.
The only metrics that matter is it done on time, on budget and meets requirements.
But if Claude Code is generating “useless code” for you, you’re doing it wrong
And I assure you that my implementations from six years of working with consulting departments/companies (including almost four as blue badge, RSU earning consultant at AWS ProServe) have never gone unused.
Re: Please do not A/B test my workflow
#148Earlier quoted context omitted.
Do you think it will reach "understanding of semantics", true cognition, within our lifetimes ? Or performance indistinguishable from that even if not truly that. Not sure. I am not so optimistic. People got intoxicated with nuclear powered cars , flying cars , bases on the moon ,etc all that technological euphoria from the 50's and 60's that never panned out. This might be like that. I think we definitely stumbled o…
Why do you think it doesn't have understanding of semantics? I think that was one of the first things to fall to LLMs, as even early models interpreted the word "crashed" differently in "I crashed my car" and "I crashed my computer", and were able to easily conquer the Winograd schema challenge.
> even early models interpreted the word "crashed" differently in "I crashed my car" and "I crashed my computer"
That has nothing to do with semantical understanding beyond word co-occurrence.
Those two phrases consistently appear in two completely different contexts with different meaning. That's how text embeddings can be created in an unsupervised way in the first place.
Re: Please do not A/B test my workflow
#149Earlier quoted context omitted.
> People don’t enjoy the thought of auditing code… someone else will do it I think that with modern LLMs auditing a big project personally, instead of relying on someone else to do it, actually became more realistic. You can ask an LLM to walk you through the code, highlight parts that seem unusual or suspicious, etc. On the other hand, LLMs also made producing code cheaper then ever, so you can argue, that big proje…
fuck man, I'm either seriously stupid or y'all are taking crazy pills. LLMs are auto-complete on steroids; I've lived through enough iterations of Markov Chains giving semi-sensible output (that we give meaning to) and neural networks which present the illusion of intelligence to see directly what these LLMs are: a fuck load of compute designed to find "the next most common word" given the preceding 10,000 or more wo…
This message contains a lot of emotions and not too many coherent arguments. What did you actually want to say?
Re: Please do not A/B test my workflow
#150Earlier quoted context omitted.
Anthropic have done a lot of things that would give me pause about trusting them in a professional context. They are anything but transparent, for example about the quota limits. Their vibe coded Claude code cli releases are a buggy mess too. Also the model quality inconsistency: before a new model release, there’s a week or two where their previous model is garbage. A/B testing is fine in itself, you need to learn a…
> vibe coded Claude code cli releases are a buggy mess too this is what gets me. are they out of money? are so desperate to penny pinch that they can't just do it properly? what's going on in this industry?
I’m a huge user of AI coding tools but I feel like there has been some kind of a zeitgeist shift in what is acceptable to release across the industry. Obviously it’s a time of incredibly rapid change and competition, but man there is some absolute garbage coming out of companies that I’d expect could do better without much effort. I find myself asking, like, did anyone even do 5 minutes of QA on this thing?? How has this major bug been around for so long?
“It’s kind of broken, maybe they will fix it at some point,” has become a common theme across products from all different players, from both a software defect and service reliability point of view.