I'm a principal engineer at Microsoft. I barely program anymore • Hacker News Board

Re: I'm a principal engineer at Microsoft. I barely program anymore

#31

Post by payneio » Thu, Oct 23, 2025, 8:39 PM UTC

Madmallard wrote: post #27

payneio wrote: post #26

Earlier quoted context omitted.

Also... "scammer and AI grifter"?? Damn dude. It's any early-stage open-source experiment result and, mostly, just talking about how it makes me question whether or not I'll be programming in the future. Nobody's asking for your money.

My last comment wasn't really directed at you it just reminded me of how I feel about the whole scene right now.

I feel that. I've been on an emotional roller-coaster for three years now. I didn't expect any of this before then. :O

Re: I'm a principal engineer at Microsoft. I barely program anymore

#32

Post by payneio » Fri, Oct 24, 2025, 5:55 AM UTC

Madmallard wrote: post #28

payneio wrote: post #25

Earlier quoted context omitted.

I get it. I've been through cycles of this over the past three years, too. Used a lot of various tools, had a lot of disappointment, wasted a lot of time and money. But this is the kinda the whole point of my post... In our system, we added fact checking itself, comparing different approaches, summarizing and effectively utilizing the "wisdom of the crowd" (and it's success over time). And it made it work massively b…

You're going to have to put quotes around "fact checking" if you're using LLMs to do it. "comparing different approaches, summarizing and effectively utilizing the "wisdom of the crowd" (and it's success over time)" I fail to see how this is defensible as well.

Compiling and evaluating output are types of fact checking. We've done more extensive automated evaluations of "groundedness" by extra ting factual statements and seeing whether or not they are based on input data or hallucinated. There are many techniques that work well.

For comparisons, you can ask the model to eval on various axis e.g. reliability, maintainability, cyclometeic complexity, API consistency, whatever, and they generally do fine.

We run multi-trial evals with multiple inputs across multiple semantic and deterministic metrics to create statistical scores we use for comparisons... basically creating benchmark suites by hand or generated. This also does well for guiding development.

Re: I'm a principal engineer at Microsoft. I barely program anymore

#33

Post by payneio » Fri, Oct 24, 2025, 6:59 PM UTC

Madmallard wrote: post #28

payneio wrote: post #25

Earlier quoted context omitted.

I get it. I've been through cycles of this over the past three years, too. Used a lot of various tools, had a lot of disappointment, wasted a lot of time and money. But this is the kinda the whole point of my post... In our system, we added fact checking itself, comparing different approaches, summarizing and effectively utilizing the "wisdom of the crowd" (and it's success over time). And it made it work massively b…

You're going to have to put quotes around "fact checking" if you're using LLMs to do it. "comparing different approaches, summarizing and effectively utilizing the "wisdom of the crowd" (and it's success over time)" I fail to see how this is defensible as well.

And by "wisdom of the croud", I'm referring to sharing what works well and what doesn't and building good approaches into the frameworks... encoding human expertise. We do it all the time.