Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude is not your architect. Stop letting it pretend

hollandtech.net

Post reply on HN
Print view View original

71–80 of 209 posts

Re: Claude is not your architect. Stop letting it pretend

#71

Post by awesome_dude »

operatingthetan wrote: post #45
awesome_dude wrote: post #39

Earlier quoted context omitted.

AI uses a high confidence tone - likely because its training data is heavy on authoritative texts/reference books. And it does get people into a lot of trouble. I have got into trouble with it when it is extremely confident about something I am not very familiar with (as recently as two weeks ago with Claude). I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition…

>And it does get people into a lot of trouble. Pretty much everyone takes it at face value unless we know otherwise from prior experience. Even the most advanced models make embarrassing mistakes and fumble with simple tasks. Yet we are very willing to give them exceptional slack for it? I wish I knew why. Are people just that easily overcome by confident voices?

Yeah - I don't know /why/ but, as I say, I've been guilty of that myself, very recently, despite knowing it's a shockingly poor guide when left to its own devices.

Maybe because when it's right it actually expands my knowledge - there have been genuine instances where it's gone - something to the effect of - "Yo, there's this other idea for approaching the problem" which has turned out to be exactly what I was looking for?

Re: Claude is not your architect. Stop letting it pretend

#72

Post by regularfry »

spacedcowboy wrote: post #20
bad_username wrote: post #9

I think the article has the correct message, but I disagree with this: > It’s just incapable of the thing that makes a real architect valuable: saying “no.” From my experience Claude is excellent at saying "no". It won't say "no" if the prompt doesn't call for it (it won't say "no" to your direct request to do something, usually). But it offers good critique and happily pushes back if you make it clear that that's a…

It actually got quite snippy with me, when I was trying to get it to debug some issues. It kept on saying that the "burn rate" wasn't progressing and "we" should refocus our efforts somewhere else. Eventually I got something like "I have told you three times now that this is not the best approach to be taking to reduce the burn-rate and you have not taken that advice". And it stopped helping out. So I was blunt, and…

That sounds more like a spec change than a set of bug fixes, even if the conclusion is that the potentially implicit spec you started with was incorrect. I've had an interesting experience extracting a spec from some existing code, making some modifications, then saying basically "implement this spec, don't come back until you're done".

An interesting experiment would be to try having the agent annotate the code with the relevant spec section while it's extracting the spec, then to have the agent update the spec with the new requirement - as an explicit change with something like "This section updated in V2 with...." - and have the agent update the codebase from that.

Some of these problems do just need breaking down a little further than you'd think to make the agent's life easier. This might be one of them.

Re: Claude is not your architect. Stop letting it pretend

#73

Post by cvwright »

retrac wrote: post #2

For fun I've been vibe coding something I know well: toolchains. Maybe not the right thing to vibe code. But I can more or less judge the quality of the output. When left to its own devices with the instructions "make an assembler for the architecture in ISA.md" -- well Claude picked Python as the implementation language. Tokens lifted through a bunch of regex. No expression parser! Oh dear. My first assembler was li…

LLMs are bringing us back to all the “proper” software engineering stuff that we’ve always known we should be doing, but until now we never had enough time/people/money to do it right.

Brainstorming and research before writing a design.

Writing a design or spec before writing the code.

Comprehensive unit tests.

Etc etc etc.

Like you, I get vastly better output from the tool when I create a detailed spec in markdown before I let it start coding. And bonus, the LLM is pretty good at helping with the spec too.

Re: Claude is not your architect. Stop letting it pretend

#74

Post by tosti »

mlsu wrote: post #51
ISL wrote: post #44

Accountability is the biggest unaddressed challenge for AI implementation. When one person is able to do too much too quickly, they can create more liability than they can accommodate if something fails. It is essential that a human is responsible for the utilization of any AI output in the real world, but that is not enough. For our own sakes, we must find ways to minimize the tech-debt bankruptcy blast-radius of th…

This is the whole point. “Sorry, the AI said that you are not approved for this cancer treatment, it’s not going to be covered.” “Sorry, the AI said that you were at the scene when the crime took place.” “Sorry, the AI has flagged your account for inappropriate content.” “Sorry, the AI says that you are too risky to lend to.” …

Computer says no, but worse.

Re: Claude is not your architect. Stop letting it pretend

#75

Post by tquinn35 »

joe_mamba wrote: post #58
tquinn35 wrote: post #31

Earlier quoted context omitted.

I agree with what you’re saying but I think the difference is many managers and above think that AI is infallible or at least much less so than it actually is and that causes problems. Everyone is aware that humans write poor code and treat the code as so. Not so with AI code. I’ve seen devs and managers cut corners in testing/reviewing code cause AI wrote it and they think it’s solid. Sure you could blame anyone cut…

> I think the difference is many managers and above think that AI is infallible Good for them. I hope they believe this because one of two things will happen. Either they win on the free market because they went all in on AI and beat their competition thanks to AI productivity increases. Or, their AI code is shit and they collapse and go bankrupt, and get beaten by the competitors using human written code so then the…

Quite an extreme view. Chances are it lands someplace in the middle.

Re: Claude is not your architect. Stop letting it pretend

#77

Post by darkteflon »

irishcoffee wrote: post #37
operatingthetan wrote: post #29

Earlier quoted context omitted.

>anthropomorphism problem. AI is a tool. It needs to be subservient. Suggesting it should be 'subservient' is also anthropomorphizing. I think your callout is correct, but you still can't help but refer to it in terms we use for other people or living entities. This is by design from the AI companies.

My drill, hammer, and chainsaw are also subservient, they just have a much cruder form of communication, noise.

I really do feel like “power tool” is the ultimate metaphor for these things. Their interface naturally confuses us into anthropomorphising them, but once you stop treating them like intelligent agents and start treating them with the same wariness, respect and intent you show to your table saw, the fun begins.

Re: Claude is not your architect. Stop letting it pretend

#78

Post by magicalhippo »

bad_username wrote: post #9

I think the article has the correct message, but I disagree with this: > It’s just incapable of the thing that makes a real architect valuable: saying “no.” From my experience Claude is excellent at saying "no". It won't say "no" if the prompt doesn't call for it (it won't say "no" to your direct request to do something, usually). But it offers good critique and happily pushes back if you make it clear that that's a…

I have it in the system/base prompt to be critical of what I say and not to assume what I say is correct or a good idea. I get push-back often from all of the three big ones.

Gemini is the most aggressive where it often picks on things if I leave out "the obvious" details, GPT somewhere inbetween, and Claude less so but still does it.

Re: Claude is not your architect. Stop letting it pretend

#79

Post by AlienRobot »

mlsu wrote: post #51
ISL wrote: post #44

Accountability is the biggest unaddressed challenge for AI implementation. When one person is able to do too much too quickly, they can create more liability than they can accommodate if something fails. It is essential that a human is responsible for the utilization of any AI output in the real world, but that is not enough. For our own sakes, we must find ways to minimize the tech-debt bankruptcy blast-radius of th…

This is the whole point. “Sorry, the AI said that you are not approved for this cancer treatment, it’s not going to be covered.” “Sorry, the AI said that you were at the scene when the crime took place.” “Sorry, the AI has flagged your account for inappropriate content.” “Sorry, the AI says that you are too risky to lend to.” …

>In The Unaccountability Machine, Dan Davies argues that organizations form “accountability sinks,” structures that absorb or obscure the consequences of a decision such that no one can be held directly accountable for it. Here’s an example: a higher up at a hospitality company decides to reduce the size of its cleaning staff, because it improves the numbers on a balance sheet somewhere. Later, you are trying to check into a room, but it’s not ready and the clerk can’t tell you when it will be; they can offer a voucher, but what you need is a room. There’s no one to call to complain, no way to communicate back to that distant leader that they’ve scotched your plans. The accountability is swallowed up into a void, lost forever.[0]

This, but web scale.

- https://aworkinglibrary.com/writing/accountability-sinks

Re: Claude is not your architect. Stop letting it pretend

#80

Post by ambicapter »

operatingthetan wrote: post #29
amarant wrote: post #18

Re: "the attaboy problem". I strongly disagree that this is a problem. What we have is a anthropomorphism problem. AI is a tool. It needs to be subservient. You actually can get it to point out issues in your design, if you just put enough humility and uncertainty in your prompt formulation, but more importantly, we have all seen that Claude makes mistakes. The title of this post is that it's a poor architect. Imagin…

>anthropomorphism problem. AI is a tool. It needs to be subservient. Suggesting it should be 'subservient' is also anthropomorphizing. I think your callout is correct, but you still can't help but refer to it in terms we use for other people or living entities. This is by design from the AI companies.

The AI should be subservient the way same way a ladder is subservient. A ladder is not a human.
Post reply on HN