211–220 of 1001 posts
I sympathise with the author being in the same boat, largely. I just want to emphasise a point... Calculators give 100% correct answers and yet we still hire accountants; for the simple fact that we don't want all to be accountants. People will hire software engineers for the simple fact that they do not want to be software engineers.
Let me just say AI is not nearly as good as the billions of dollars in marketing spend say.
We are months away from catastrophic bed shitting and the tech industry will pay the piper.
This feels fake/engineered but regardless of that also redundant. It's the exact same story that we've heard countless times by now. Hosted on a blog with just a single post. Named in a way that suggests that said blog was created for this very single post. What is there to learn from this other than LLMs seem to be bad for some people's psyches and that AI companies need these very stories to not get their funding s…
Don’t sell yourself short! Taste is not promptable, I suspect good taste is AGI-complete.
Especially in domains like fintech, there is a lot of accumulated wisdom, and that is what you’ll be handsomely paid for (for at least the next couple years :/ )
For example, architectural patterns, when you need bitemporality, immutable logs, CQRS, all these good patterns that can only be learned by owning years of system architecture - none of these feedback loops are in the training set.
And from a product design side, agents will just miss key concepts and you need a few words to prompt a fix - but that might represent a massive tree search optimization, or the agent on many cases would just fail to identify the requirement. These small steers feel small, but by evaporation our work has distilled down to just the extremely high value insights.
METR task time is still at weeks, doubling every 7 months; it’s years (assuming we keep riding this crazy exponential) until you hit multi-year tasks. I don’t see wisdom / Métis being solved in 2027.
All this said - I think it’s important to extrapolate forwards, if the trend continues, this will may all be true in 3-5 years. Now is the time to pre-register what metrics would make you worried, so that you can define your red lines. There will be a rapid consolidation of power and wealth if these tools continue on their existing growth trajectory.
Earlier quoted context omitted.
funny i was able to do all my taxes this year with ai help and not needing accountant.
Lol thats brave on your part, given that a mistake can cost thousands and you have no accountability (punch!) from an LLM
I wouldn't say it's particularly brave, in fact LLMs are probably better at identifying mistakes than most tax payers. The % of Americans using a CPA to file taxes is fairly small.
Earlier quoted context omitted.
It's not so much about velocity or quality, both of which LLM do (or will) provide. The real question is about accountability and liability. When a major data leak is going to happen, who will they sue or fire ? That is the value engineers provide. They understand, confirm, and take ownership.
> When a major data leak is going to happen, who will they sue or fire ? That is the value engineers provide. They understand, confirm, and take ownership. This goes for serious incidents, disasters, outages and security breaches. If there was an investigation and the answer was "a piece of software was vibe coded with AI" why would anyone trust the software vendor after that?
Even Solarwinds is still alive.
Earlier quoted context omitted.
Posting this under a burner so I don't dox myself: I work in FinTech on a regulated product. We have access to Mythos. Mythos identified part of our codebase that it confidently asserted was not complaint with a particular regulation and we were at grave risk by allowing it to operate the way it was. Except this was not the case, it had of course hallucinated what the regulation actually required (I know this because…
what am i missing? you take a spec and create tests, every little thing you use another ai to verify these tests against the spec you review the tests vs the spec (at one point human review) you put the tests off limits to change / wall them. you let the ai write the software that fulfills the tests. there will be some gaps where you repeat the cycle above if the tests fulfill the spec, the code will fulfill the spec
The only way to test this is to test it out, in real life. Sometimes people see results, sometimes people don't. Note that yes, I am including the entire iteration process - even after iterating, people still don't see results with AI.
I have had both positive and negative experiences with AI, over multi-week projects. But apparently on hackernews, anything positive about AI is proof that AI is superhuman and taking over, and all follies about AI are lies by stupid humans who secretly have psychological dispositions to fear AI. Sometimes the AI genuinely isn't good enough. Are we not allowed to say that now? We might not know why, but it's just the truth.
The other solution is to formally analyze the entire space of possible actions the agent can take a priori. Then yes, you can definitively say whether or not the principle breaks or not. Can you, though? Can you give a formal specification for the space of possible actions for AI and show that your loop never breaks, or breaks less than humans, or any other sensible criteria? If not, then you can't just give an abstract principle and start making inferences from that.
Earlier quoted context omitted.
Which other profession has the same amount of training data freely available for the taking?
Don't need much training data for bank/insurance/retail analyst work - it's just basic reasoning and data retrieval. If AI could crack the programming nut - one of the most intellectually challenging professions - it can handle the rest with ease. The only human role will be high level monitoring - and even this will be largely automated so fewer will be required.
> I still have one pillar standing, though: code quality and software architecture - what's now being reduced to being called "taste". Genuine question: what exactly is "quality"? It's something I've been trying to understand for a very long time. It seems like it's entirely contextual, and it has both subjective and objective facets (the latter only for quantifiable things, and still entirely contextual).
Maybe ask the same question about other things. What makes a good guitar? What makes a good chair? What makes a good airplane? What makes a good book? What makes a good song? What makes good art? Each of these has a long list of very specific goals and concerns. And to help define the boundaries, also ask what makes something bad, and what makes something mediocre.
Code quality starts with functionality. Does it perform the stated requirements? Does it have testing in place to catch breaking changes in functional requirements? That’s the basic stuff that probably isn’t part of “taste”. A lot of code quality goals center around how code changes over time, and beliefs about designing to avoid functional breakage.
For example you can ask things like does the code use minimal dependencies? Is the code organized into clean classes/modules/functions that each have a single clear role? Is the API easy to read, understand, and use? Is the API hard to misuse accidentally? Is all the code easy to read? Is there documentation, and is the documentation useful, and more than a list of contents? Is the code self-documenting? Is the code efficient, both in how it executes, and in its use of code itself? Is the code designed so that it won’t fail when someone runs it with different sized types, or a different compiler or execution environment, or on a different architecture? Is the code surprisingly elegant and fun to use?
Those are just the beginning. There are of course more layers of application-specific and environment-specific and audience-specific qualities. The good news is that quality depends on your own goals, you can decide which aspects of taste matter to you, and ignore the ones that don’t. It’s fine if your taste & goals change over time.
Wut? I pilot LLMs all day but there's no way in hell I'd agree to be at the helm of a finance product. That first pillar is still there. Maybe the author isn't aware of the impact they have, but I know, with the evidence of reverted PRs, that when I step outside my area of deep knowledge I can no longer call BS on the agents. Our most capable agent, with access to the same kind of distributed systems the author talks…