Superpowers: How I'm using coding agents in October 2025 • Hacker News Board

Re: Superpowers: How I'm using coding agents in October 2025

#51

Post by lcnPylGDnU4H9OF » Sat, Oct 11, 2025, 4:15 PM UTC

j_bum wrote: post #45

simonw wrote: post #31

Earlier quoted context omitted.

Here's one from today: https://mitchellh.com/writing/non-trivial-vibing

This was a fun read. I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc. I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work. This has been working very well for me.

Do you mind linking to example spec/to-do files?

Re: Superpowers: How I'm using coding agents in October 2025

#52

Post by lcnPylGDnU4H9OF » Sat, Oct 11, 2025, 4:16 PM UTC

spankibalt wrote: post #41

Avicebron wrote: post #7

I often feel these types of blogposts would be more helpful if they demonstrated someone using the tools to build something non-trivial. Is Claude really "learning new skills" when you feed it a book, or does it present it like that because you're prompting encourages that sort of response-behavior. I feel like it has to demo Claude with the new skills and Claude without. Maybe I'm a curmudgeon but most of these type…

> "Maybe I'm a curmudgeon but most of these types of blogs feel like marketing pieces with the important bit is that so much is left unsaid and not shown, that it comes off like a kid trying to hype up their own work without the benefit of nuance or depth." C'mon, such self-congratulatory "Look at My Potency: How I'm using Nicknack.exe" fluffies always were and always will be a staple of the IT industry.

Still, the best such pieces are detailed and explanatory.

Re: Superpowers: How I'm using coding agents in October 2025

#53

Post by theptip » Sat, Oct 11, 2025, 4:24 PM UTC

> some of the ones I've played with come from telling Claude "Here's my copy of programming book. Please read the book and pull out reusable skills that weren't obvious to you before you started reading

This is actually a really cool idea. I think a lot of the good scaffolding right now is things like “use TDD” bit if you link citations to the book, then it can perhaps extract more relevant wisdom and context (just like I would by reading the book), weather than using the generic averaged interpretation of TDD derived from the internet.

I do like the idea of giving your Claude a reading list and some spare tokens on the weekend where you’re not working, and having it explore new ideas and techniques to bring back to your common CLAUDE.md.

Re: Superpowers: How I'm using coding agents in October 2025

#54

Post by potatolicious » Sat, Oct 11, 2025, 4:28 PM UTC

mwigdahl wrote: post #22

khaledh wrote: post #9

Earlier quoted context omitted.

Agreed. The methodology needed here is something like an A/B test, with quantifiable metrics that demonstrate the effectiveness of the tool. And to do it not just once, but many times under different scenarios so that it demonstrates statistical significance. The most challenging part when working with coding agents is that they seem to do well initially on a small code base with low complexity. Once the codebase get…

The problem is that you're talking about a multistep process where each step beyond the first depends on the particular path the agent starts down, along with human input that's going to vary at each step. I made a crude first stab at an approach that at least uses similar steps and structure to compare the effectiveness of AI agents. My approach was used on a small toy problem, but one that was complex enough the ag…

What you're getting at is the heart of the problem with the LLM hype train though, isn't it?

"We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought.

But in the realm of LLM-enabled use cases they're also expensive. You'd need to recruit dozens, perhaps even hundreds of developers to do this, with extensive observation and rating of the results.

So rather than actually try to measure the efficacy, we just get blog posts with cherry-picked example of "LLM does something cool". Everything is just anecdata.

This is also the biggest barrier to actual LLM adoption for many, many applications. The gap between "it does something REALLY IMPRESSIVE 40% of the time and shits the bed otherwise" and "production system" is a yawning chasm.

Re: Superpowers: How I'm using coding agents in October 2025

#55

Post by zahlman » Sat, Oct 11, 2025, 4:29 PM UTC

> It also bakes in the brainstorm -> plan -> implement workflow I've already written about. The biggest change is that you no longer need to run a command or paste in a prompt. If Claude thinks you're trying to start a project or task, it should default into talking through a plan with you before it starts down the path of implementation.

... So, we're refactoring the process of prompting?

> As Claude and I build new skills, one of the things I ask it to do is to "test" the skills on a set of subagents to ensure that the skills were comprehensible, complete, and that the subagents would comply with them. (Claude now thinks of this as TDD for skills and uses its RED/GREEN TDD skill as part of the skill creation skill.)

> The first time we played this game, Claude told me that the subagents had gotten a perfect score. After a bit of prodding, I discovered that Claude was quizzing the subagents like they were on a gameshow. This was less than useful. I asked to switch to realistic scenarios that put pressure on the agents, to better simulate what they might actually do.

... and debugging it?

... How many other basic techniques of SWEng will be rediscovered for the English programming language?

Re: Superpowers: How I'm using coding agents in October 2025

#56

Post by philbo » Sat, Oct 11, 2025, 4:31 PM UTC

spprashant wrote: post #26

I am not ashamed to admit this whole agentic coding movement has moved beyond me. Not only do I have know everything about the code, data and domain, but now I need to understand this whole AI system which is a meta skill of its own. I fear I may never be able catch up till someone comes along and simplifies it for pleb consumption.

I think this and other recent posts here hugely overcomplicate matters. I notice none of them provides an A/B test for each item of complexity they introduce, there's just a handwavy "this has proved to work over time".

I've found that a single CLAUDE.md does really well at guiding it how I want it to behave. For me that's making it take small steps and stop to ask me questions frequently, so it's more like we're pairing than I'm sending it off solo to work on a task. I'm sure that's not to everyone's taste but it works for me (and I say this as someone who was an agent-sceptic until quite recently).

Fwiw my ~/.claude/CLAUDE.md is 2.2K / 49 lines.

Re: Superpowers: How I'm using coding agents in October 2025

#57

Post by nightski » Sat, Oct 11, 2025, 4:31 PM UTC

simonw wrote: post #31

Avicebron wrote: post #7

I often feel these types of blogposts would be more helpful if they demonstrated someone using the tools to build something non-trivial. Is Claude really "learning new skills" when you feed it a book, or does it present it like that because you're prompting encourages that sort of response-behavior. I feel like it has to demo Claude with the new skills and Claude without. Maybe I'm a curmudgeon but most of these type…

Here's one from today: https://mitchellh.com/writing/non-trivial-vibing

Even though the author refers to it as "non-trivial", and I can see why that conclusion is made, I would argue it is in fact trivial. There's very little domain specific knowledge needed, this is purely a technical exercise integrating with existing libraries for which there is ample documentation online. In addition, it is a relatively isolated feature in the app.

On top of that, it doesn't sound enjoyable. Anti slop sessions? Seriously?

Lastly, the largest problem I have with LLMs is that they are seemingly incapable of stopping to ask clarifying questions. This is because they do not have a true model of what is going on. Instead they truly are next token generators. A software engineer would never just slop out an entire feature based on the first discussion with a stakeholder and then expect the stakeholder to continuously refine their statement until the right thing is slopped out. That's just not how it works and it makes very little sense.

Re: Superpowers: How I'm using coding agents in October 2025

#58

Post by causal » Sat, Oct 11, 2025, 4:37 PM UTC

Avicebron wrote: post #7

I often feel these types of blogposts would be more helpful if they demonstrated someone using the tools to build something non-trivial. Is Claude really "learning new skills" when you feed it a book, or does it present it like that because you're prompting encourages that sort of response-behavior. I feel like it has to demo Claude with the new skills and Claude without. Maybe I'm a curmudgeon but most of these type…

Using LLMs for coding complex projects at scale over a long time is really challenging! This is partly because defining requirements alone is much more challenging than most people want to believe. LLMs accelerate any move in the wrong direction.

Re: Superpowers: How I'm using coding agents in October 2025

#59

Post by troupo » Sat, Oct 11, 2025, 4:38 PM UTC

simonw wrote: post #21

I can't recommend this post strongly enough. The way Jesse is using these tools is wildly more ambitious than most other people. Spend some time digging around in his https://github.com/obra/Superpowers repo. I wrote some notes on this last night: https://simonwillison.net/2025/Oct/10/superpowers/

This looks like usage rules in Elixir, but for agent behaviors, and currently specifically for Claude: https://hexdocs.pm/usage_rules/readme.html

Re: Superpowers: How I'm using coding agents in October 2025

#60

Post by jelling » Sat, Oct 11, 2025, 4:39 PM UTC

jackblemming wrote: post #11

Seems cute, but ultimately not very valuable without benchmarks or some kind of evaluation. For all I know, this could make Claude worse.

Same. We've all fooled ourselves into believing that an LLM / stochastic process was finally solved based on a good result. But the sample size is always to low to be meaningful.