Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can't unit test for taste

dev.karltryggvason.com

Post reply on HN
Print view View original

21–30 of 157 posts

Re: You can't unit test for taste

#21

Post by dasil003 »

chantepierre wrote: post #8
another-dave wrote: post #5

Earlier quoted context omitted.

"The effort length and instantaneous power output changes, of course." but that's what the phrase is meant to convey, right? Don't run through consumable X (energy/money/etc) like there's no tomorrow - even though there's now, we've got dozens more of those that we need to meet, so you're better off getting this one done at 75% than committing 100% to it and failing on all the others.

Yeah you're right, I hear it more like "this is a week long hike, not a sprint" as if a marathon included rest. In any length of racing there's no tomorrow. But I'm doing tongue-in-cheek pedanticness here and will stop that right now !

I'd wager that if a manager says that they want you to take it more like a real marathon and less like long hike.

Re: You can't unit test for taste

#22

Post by trjordan »

delichon wrote: post #17
trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

You may be able to effectively externalize taste by "hot or not" style pair testing. Enough comparisons and I'd expect ML to be able to mimic human taste by latching on to features we're not well aware of influencing us.

This is RL, right? Like, this is exactly why models have mostly converged around obvious style, because we train them literally on thumbs-up/thumbs-down data of what good behavior and good code looks like.

And that's why it's so hard to get a model to reproduce the specific taste of a person or an organization. My taste is different than yours, so if we dump our aggregate preferences into RL, in averages out to nothing interesting.

For the code-writing case, this means you end up reviewing every line of code, looking for places where you'd thumbs-down the code. Not every line of code contains a real decision, though, so it feels like a waste of time.

Re: You can't unit test for taste

#23

Post by sesm »

timroman wrote: post #19

https://pureinference.com/insights/taste-is-the-new-skill I wrote about this a few months back. Rick Rubin is famous for this. I do think it is something that can be trained though, it just needs a lot more context. Taste builds over time through lots of unit tests, through lots of content writing, through an accumulation of product decisions. It’s hard to put it in the individual spec, but it can be teased out of 10…

> Rick Rubin told Anderson Cooper he has no technical ability. Doesn't play instruments. Can't work a mixing board.

If you watch his interview on Rick Beato's channel, this myth will fall apart. He plays guitar, had his own punk rock band and his guitar playing is featured on some high-profile records he produced. Also, he has a lot of practical experience with all kinds of studio equipment.

Re: You can't unit test for taste

#24

Post by paytonjjones »

trjordan wrote: post #22
delichon wrote: post #17

Earlier quoted context omitted.

You may be able to effectively externalize taste by "hot or not" style pair testing. Enough comparisons and I'd expect ML to be able to mimic human taste by latching on to features we're not well aware of influencing us.

This is RL, right? Like, this is exactly why models have mostly converged around obvious style, because we train them literally on thumbs-up/thumbs-down data of what good behavior and good code looks like. And that's why it's so hard to get a model to reproduce the specific taste of a person or an organization. My taste is different than yours, so if we dump our aggregate preferences into RL, in averages out to nothi…

This is, in short, the big current problem with AI.

LLMs are built for scale so they've given up on the kind of online learning / "long term memory" processes that would individualize them.

The LLM is permanently locked to being a really cracked engineer on their first day at your company, looking at your codebase for the first time.

You can scaffold a bit with .md files, but at the moment they lack the ability to do what humans do: go to sleep, encode things from short to long term memory, and wake up the next day with more specific knowledge baked in.

Re: You can't unit test for taste

#25

Post by timroman »

sesm wrote: post #23
timroman wrote: post #19

https://pureinference.com/insights/taste-is-the-new-skill I wrote about this a few months back. Rick Rubin is famous for this. I do think it is something that can be trained though, it just needs a lot more context. Taste builds over time through lots of unit tests, through lots of content writing, through an accumulation of product decisions. It’s hard to put it in the individual spec, but it can be teased out of 10…

> Rick Rubin told Anderson Cooper he has no technical ability. Doesn't play instruments. Can't work a mixing board. If you watch his interview on Rick Beato's channel, this myth will fall apart. He plays guitar, had his own punk rock band and his guitar playing is featured on some high-profile records he produced. Also, he has a lot of practical experience with all kinds of studio equipment.

That’s exactly it. His taste isn’t in any one thing. It’s the esoteric and accumulated from a variety of things. You can’t package it up. That’s the point on the project specs. I can never get it right in one, but the arc over 100 becomes visible. Especially to an LLM that has the capacity to intake and understand that.

Re: You can't unit test for taste

#26

Post by giancarlostoro »

trjordan wrote: post #4

You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can. Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want. Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I…

What's kind of funny is this is how I implemented "gates" for the ticketing system I built for Claude, because Beads would just close tickets without validation. I have tickets that are literally "Human validation" tier, so it will work on the next available thing until I personally tell the model to close it. So, in that spirit, yeah, you can unit test for taste, if you implement external validation.

Unit test runs, waits for human input before passing or failing, which might seem out of the norm, but we already have QA do manual testing.

Re: You can't unit test for taste

#27

Post by pjmlp »

Exactly one of the reasons I never went down with all the TDD dogma of only writing code to fix broken tests.

There is a reason conference talks are always about plain algorithms and data structures.

Re: You can't unit test for taste

#28

Post by trjordan »

paytonjjones wrote: post #24
trjordan wrote: post #22

Earlier quoted context omitted.

This is RL, right? Like, this is exactly why models have mostly converged around obvious style, because we train them literally on thumbs-up/thumbs-down data of what good behavior and good code looks like. And that's why it's so hard to get a model to reproduce the specific taste of a person or an organization. My taste is different than yours, so if we dump our aggregate preferences into RL, in averages out to nothi…

This is, in short, the big current problem with AI. LLMs are built for scale so they've given up on the kind of online learning / "long term memory" processes that would individualize them. The LLM is permanently locked to being a really cracked engineer on their first day at your company, looking at your codebase for the first time. You can scaffold a bit with .md files, but at the moment they lack the ability to do…

100%. The problem with them isn't making sure they're doing the right thing, it's making sure they're not making bad assumptions.

IMHO this is where code review goes until we fix the individualized model thing: you need to review the decisions the agent made, where you didn't steer. Most will be right. A few will be disastrously wrong. But decision-by-decision is a lot less to review than line-by-line of code.

Re: You can't unit test for taste

#29

Post by plastic-enjoyer »

paytonjjones wrote: post #24
trjordan wrote: post #22

Earlier quoted context omitted.

This is RL, right? Like, this is exactly why models have mostly converged around obvious style, because we train them literally on thumbs-up/thumbs-down data of what good behavior and good code looks like. And that's why it's so hard to get a model to reproduce the specific taste of a person or an organization. My taste is different than yours, so if we dump our aggregate preferences into RL, in averages out to nothi…

This is, in short, the big current problem with AI. LLMs are built for scale so they've given up on the kind of online learning / "long term memory" processes that would individualize them. The LLM is permanently locked to being a really cracked engineer on their first day at your company, looking at your codebase for the first time. You can scaffold a bit with .md files, but at the moment they lack the ability to do…

> LLMs are built for scale so they've given up on the kind of online learning / "long term memory" processes that would individualize them.

I wonder if this is even desirable from a product perspective. You probably don't want online learning in a product that you are selling because you can't guarantee a consistent quality of the product.

Re: You can't unit test for taste

#30

Post by voidUpdate »

throw93949444 wrote: post #2

> For example, my native Iceland had a nice mix of nature, historical sites and populated places. You absolutely can unit test for taste, just put an agent into loop, and write into prompt what you like. Then do scoring... Iceland is really bad example, it basically has one populated site (capital) and circular road that goes around the island.

I'm pretty sure there's more points of interest in the entirety of Iceland than just Reykjavík and Route Number One
Post reply on HN