Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can't unit test for taste

dev.karltryggvason.com

Post reply on HN
Print view View original

41–50 of 157 posts

Re: You can't unit test for taste

#41

Post by pjmlp »

bob1029 wrote: post #37
pjmlp wrote: post #27

Exactly one of the reasons I never went down with all the TDD dogma of only writing code to fix broken tests. There is a reason conference talks are always about plain algorithms and data structures.

The biggest flaw I've seen with TDD is the fact that correctness does not compose upward. Every time two units come into contact, you've got an entirely new kind of unit. The tests from constituents do not cover emergent properties of the new things. You will repeat this same exercise the entire way up to the top, and the moment you come into contact with the customer (they want to change everything), the house of ca…

Exactly, the whole system thinking and large scale architecture also fails apart, when writing everything from little working tests.

Re: You can't unit test for taste

#42

Post by Dumblydorr »

trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

Randomized trial. Half of them pledge to use AI freely and liberally, half of them to never use it, compare via surveys and off-AI tests after X months. Could even flip it so then the non-users used it for X months and vice versa, see if losses/gains are stable.

Re: You can't unit test for taste

#45

Post by dyarosla »

pydry wrote: post #36
trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

I remember reading an interview with a fireman who described a time when his buddy evacuated a team because he "felt" that a floor would collapse imminently. He couldn't articulate why but they trusted his gut and it did collapse. A lot of software engineering relies on that kind of intuition and on a good team you can integrate it and benefit from it and avoid all manner of floor collapses.

To play devil’s advocate, intuition is still a physical response to stimuli mixed with knowledge of past experience. Hypothetically it could be modeled- the problem here comes down to how to encode it.

Re: You can't unit test for taste

#46

Post by al_borland »

delichon wrote: post #17
trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

You may be able to effectively externalize taste by "hot or not" style pair testing. Enough comparisons and I'd expect ML to be able to mimic human taste by latching on to features we're not well aware of influencing us.

Wouldn't this style of training suffer from the AI learning things the user didn't intend? I may thumbs down something for a specific detail I don't like, while other things in it are great. Certain traits that tend to occur together go along for the ride. We see similar things happen in natural selection, where mates may be chosen for 1 specific feature, and other less desirable things come along for the ride.

Outside of AI, I run into this issue when taking basic personality tests. A question may be written for a specific reason, which influences the results, but the reason for my answer may be completely unrelated to the reason intended by the person who made the test.

Re: You can't unit test for taste

#48

Post by eithed »

trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

I agree and indeed externalize everything you know *that matters*.

Want to follow certain pattern, or convention - define it, ie active record vs repository pattern, stick is as an ADR! You don't know what you want? Look at what Claude produces and then acquire taste, mark this as convetion that future sessions will follow, but stick to *one* convention!

Treat your LLMs as junior developers willing to apply various patterns willy nilly, caring only about fulfilling the ACs of given task and not about the longevity or well being of the system in general. They will not look at bigger picture to check if given pattern applies globally, or even if there are any other patterns.

Re: You can't unit test for taste

#49

Post by deadbabe »

trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

You cannot externalize taste. You could perhaps mimic someone’s taste, but that’s not the taste. Knowing the taste requires actually tasting it. You can’t capture the taste, it’s already gone.

Re: You can't unit test for taste

#50

Post by eithed »

trjordan wrote: post #22
delichon wrote: post #17

Earlier quoted context omitted.

You may be able to effectively externalize taste by "hot or not" style pair testing. Enough comparisons and I'd expect ML to be able to mimic human taste by latching on to features we're not well aware of influencing us.

This is RL, right? Like, this is exactly why models have mostly converged around obvious style, because we train them literally on thumbs-up/thumbs-down data of what good behavior and good code looks like. And that's why it's so hard to get a model to reproduce the specific taste of a person or an organization. My taste is different than yours, so if we dump our aggregate preferences into RL, in averages out to nothi…

Yes and no.

If I were to ask you - what convention you want to follow for your database columns - camelcase or snakecase? There's no correct global answer. There's no overarching truth that should apply to all databases in existence (even if you'll focus on a certain type of database). Hence the no.

But yes, because in the context of existing system there is a convention. If it's snakecase, you create new tables with snakecase column names.

LLMs will generally follow conventions, but sometimes they will not, because indeed - global truths (or at least, the "last article it read" truths) sometimes win over (I assume)

Post reply on HN