Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We ran Anthropic’s interviews through structured LLM analysis

playbookatlas.com

Post reply on HN
Print view View original

61–70 of 91 posts

Re: We ran Anthropic’s interviews through structured LLM analysis

#61

Post by userbinator »

jp8585 wrote: post #28
malfist wrote: post #23

This article is rife with unedited llm signals. This makes me question their methodology here. I want you believe what they found, but I don't trust this analysis. If they were this sloppy with the write up, how sloppy were they with the science?

We have a full page on the methodology we used! Let me know if you’d like access to the dataset we created for this. The aim was not to be scientific but to flush out some deeper meanings from these interviews that typical nlp techniques struggle with. Ps: Of course we used llm tools as a writing aid, I’d be willing to bet those “signals” probably come from my own writing though and my appreciation of Tom Wolfe. I’ve…

We have a full page on the methodology we used! Let me know if you’d like access to the dataset we created for this.

I'm not sure if you realise that those two sentences sound like 100% verbatim LLM output, or am I actually replying to a bot and not a human.

Re: We ran Anthropic’s interviews through structured LLM analysis

#62

Post by userbinator »

layer8 wrote: post #18
huevosabio wrote: post #12

``` Creatives have the highest struggle scores and the highest adoption rates. ``` Here is my guess for the puzzle: creative work is subjective and full of scaffolding. AI can easily generate this subjective scaffolding to a "good enough" level so it can get used without much scrutiny. This is very attractive for a creative to use on a day to day basis. But, given the amount of content that wasn't created by the crea…

One issue with AI for creatives is that it’s virtually impossible to get AI to create a specific vision you have in mind. It creates something , but you just have to accept whatever that is, you can only steer it very roughly. It can be useful for getting inspiration, but not for getting exact results. If AI was better suited for realizing one’s own creative vision and working in a detail-oriented fashion, creators w…

Having tried some AI image generation, it feels more like gambling than work --- repeatedly submitting and hoping you get the result you wanted is extremely reminiscent of pulling a one-armed bandit hoping to win, except perhaps a bit cheaper. I can certainly understand a potential for addiction though.

Re: We ran Anthropic’s interviews through structured LLM analysis

#63

Post by nl »

delusional wrote: post #25
jp8585 wrote: post #6

Earlier quoted context omitted.

I actually think things improved substantially when compared to last year. The latest batch of sota models is incredible (just ask any software engineer about what’s happening to their profession). It’s only a matter of time until other knowledge workers start getting the asphyxiating “vibe” coding treatment and that drama is what really fascinates me. People are absolutely torn. It seems that ai usage starts as a cl…

> just ask any software engineer about what’s happening to their profession I'm a professional developer, and nothing interesting is happening to the field. The people doing AI coding were already the weakest participants, and have not gained anything from it, except maybe optics. The thing that's suffocating is the economics. The entire economy has turned its back on actual value in pursuit of silicon valley smoke.

I'm a decent dev and I'm possibly 100 times as productive using AI.

It lets me concentrate on the parts I'm good at and ignore things I don't care about. Claude is a lot better at React than I am and I don't care.

Re: We ran Anthropic’s interviews through structured LLM analysis

#64

Post by travisgriggs »

How do I know this fine article wasn’t the result of

“Create a web page infographic report that is convincing and boils down the essential truths of how people are feeling about AI in different professions and domains.. Include statistics and numbers and some rolling/animated sound bite quotes.”

Re: We ran Anthropic’s interviews through structured LLM analysis

#65

Post by godelski »

wongarsu wrote: post #44
Terretta wrote: post #13

Earlier quoted context omitted.

There’s a sense of dread that comes from realizing that it’s not useful to “do work” anymore. That in order to thrive now, we need to outsource as much of your thinking to GPT as possible. If your sense of identity comes from “pure” intellectual pursuits, you are gonna have a bad time. This is 180 degrees from how to think about it. The more thinking you do as ratio to less toil, the better. The more time to apply yo…

The catch is that many professional environments have evolved values that above a certain quality floor reward quantity over quality. Even more so in the US where pointless torment is "work ethic" and pausing to think something through is "lazy" (see Bill Gate's famous quote about hiring lazy people, or "work smarter, not harder" almost being a rebel motto). Granted, that's not everywhere. There are absolutely places…

  > see Bill Gate's famous quote about hiring lazy people
I think this is part of why all this is so contentious. There's been a huge culture shift over the last decade and AI is really just a catalyst to it. We went from managers needing to stop engineers from using too much abstraction and optimizing what doesn't need to be optimized to the engineers themselves attacking abstraction. Just look how people turn Knuth's "premature optimization is the root of evil" went from "get a profiler before you optimize" to "optimization? Are you crazy?"

Fewer and fewer people I know are actually passionate about programming and it's not uncommon to see people be burned out and just want to do their 9-5. And I see a strong correlation with these people embracing AI. It makes sense if you don't care and are just trying to get the job done. I don't think it's surprising things are getting buggier and innovation slowed. We killed the passion and tried to turn it into a mechanical endeavor. It's a negative feedback loop

Re: We ran Anthropic’s interviews through structured LLM analysis

#66

Post by zdragnar »

pertymcpert wrote: post #60
Terretta wrote: post #13

Earlier quoted context omitted.

There’s a sense of dread that comes from realizing that it’s not useful to “do work” anymore. That in order to thrive now, we need to outsource as much of your thinking to GPT as possible. If your sense of identity comes from “pure” intellectual pursuits, you are gonna have a bad time. This is 180 degrees from how to think about it. The more thinking you do as ratio to less toil, the better. The more time to apply yo…

I don't necessarily agree with you completely, but I think that's a really great analogy. At the very least full of optimism.

It's a fundamentally flawed analogy. Leo's apprentices learned and improved. They studied under a master and faced serious repercussions if they bullshitted about their ability or what they had accompolished.

LLM capabilities are tied to their model, and won't improve on their own. You learn the quirks of prompting them, but they have fixed levels of skill. They don't lie, because they don't understand concepts such as truth or deception, but that means they'll spout bullshit and it's up to you to review everything with a skeptical eye.

In this analogy, you aren't the master, you're one part client demanding work, one part the janitor cleaning up after their mistakes.

Re: We ran Anthropic’s interviews through structured LLM analysis

#67

Post by colonCapitalDee »

travisgriggs wrote: post #64

How do I know this fine article wasn’t the result of “Create a web page infographic report that is convincing and boils down the essential truths of how people are feeling about AI in different professions and domains.. Include statistics and numbers and some rolling/animated sound bite quotes.”

It certainly reads like it

Re: We ran Anthropic’s interviews through structured LLM analysis

#68

Post by latentsea »

jondwillis wrote: post #56
latentsea wrote: post #45

Earlier quoted context omitted.

Nothing interesting happening in the field? If you've been paying attention the trend over the last two years has been that the problem space that requires humans to solve has been shrinking. It's continuing to shrink. That's interesting. Significantly interesting. As an engineer that's lead multiple teams including one at a world leading SaaS company, I don't consider myself one of the weakest participants in the fi…

So what does that amount to? Shared Claude code hooks and skills?

Things like that are only part of it. You can also also up your agents batting average by finding ways to build guardrails and things that inject the right context at the right time.

Like for instance we have a task runner in our project that provides a central point to do all manner of things like linting, building, testing, local deployment etc. The build, lint and test tasks are shared between local development and CI. The test tasks run the tests, take the TRX files and use a library to parse it to produce a report. So the agent can easily get access to the same info as CI is putting out about test failures. The various different test suites output reports under a consistent folder structure, they also write logs to disk under a consistent folder structure too. On failure the test tasks output a message to look at the detailed test reports and cross-reference that with the logs to debug the issue. Where possible the test reports contain correlation IDs inlined into the report.

With the above system when the agent is working through implementing something and the tests don't pass, it naturally winds up inspecting the test reports, cross referencing that with the logs, and solving the problems at a higher rate than compared to just taking a wild guess at how to run the tests and then do something random.

Getting it to write it's own guardrails by creating Roslyn Analyzers to make the build fail when it deviates from the project architecture and conventions has been another big win.

Tonnes of small things like that start to add up.

Next on my list is getting a debug MCP server, so it can set breakpoints and step through code etc.

Re: We ran Anthropic’s interviews through structured LLM analysis

#69

Post by delusional »

jp8585 wrote: post #32
delusional wrote: post #25

Earlier quoted context omitted.

> just ask any software engineer about what’s happening to their profession I'm a professional developer, and nothing interesting is happening to the field. The people doing AI coding were already the weakest participants, and have not gained anything from it, except maybe optics. The thing that's suffocating is the economics. The entire economy has turned its back on actual value in pursuit of silicon valley smoke.

That’s fascinating. If you don’t mind me asking, what type of software development do you do? Have you tried any of the latest coding tools? Or even used LLMs as a replacement for stack overflow?

Professionally, I do banking. It's a lot of integration work, sprinkled with a little algorithm every now and then. Lately I've been on capital requirements. The core of that is a system called AxiomSL, which is quite a lot of work for one guy to keep running.

In my spare time I write some algorithmic C, you can check that stuff out on github (https://github.com/DelusionalLogic) if you're curious.

I was an early adoter of LLM's. I used to lurk in the old EleutherAI discord and monitor their progress in reconstructing GPT-2 (I recall it being called GPT-J). I also played around a bunch with image generation. At this point nobody really tried applying them to code. We were just fascinated that it wrote back at all.

I have tried most of the modern models for development. I find then to generate a lot of nonsensical and unexplainable code. I've had no success (in the 30 or so times I've tried) at getting any of the models to debug or develop even small features. They usually get lost in some "best practice" and start looping on that forever. They're also constantly breaking style and violating module boundaries.

If i use them to generate documentation I find it to be surface level and repetitive. It'll make a lot of text about structures that are obvious to me just glancing at the code, but will (obviously) not have any context about the thought process that created that code, which is the only part I care about. I can read the code just fine myself. This is the same problem I find in commit messages generated with AI tools.

For the reversing I also do, I find the models to be too imprecise. It'll take large logical leaps that ruin understanding of the code I'm trying to understand. This is the only place I actually believe a properly trained (not a chatbot) model could actually succeed past the state of the art.

I don't really use stackoverflow either, I don't trust its accuracy, and it's easy to get cargo culted in software. I generally try to find my answers in official documentation, and if I can't get that I'll read the source code. If that's unavailable I'll take a guess, or reverse the thing If it's really important to me.

Re: We ran Anthropic’s interviews through structured LLM analysis

#70

Post by delusional »

nl wrote: post #63
delusional wrote: post #25

Earlier quoted context omitted.

> just ask any software engineer about what’s happening to their profession I'm a professional developer, and nothing interesting is happening to the field. The people doing AI coding were already the weakest participants, and have not gained anything from it, except maybe optics. The thing that's suffocating is the economics. The entire economy has turned its back on actual value in pursuit of silicon valley smoke.

I'm a decent dev and I'm possibly 100 times as productive using AI. It lets me concentrate on the parts I'm good at and ignore things I don't care about. Claude is a lot better at React than I am and I don't care.

100 times? You do in a day what used to take you 3 months?

Those are just not realistic numbers.

Post reply on HN