Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Future of AI Software Development

martinfowler.com

Post reply on HN
Print view View original

121–130 of 156 posts

Re: The Future of AI Software Development

#121

Post by lambda »

charcircuit wrote: post #66
ethin wrote: post #8

Earlier quoted context omitted.

How exactly can you "mitigate" prompt injections? Given that the language space is for all intents and purposes infinite, and given that you can even circumvent these by putting your injections in hex or base64 or whatever? Like I just don't see how one can truly mitigate these when there are infinite ways of writing something in natural language, and that's before we consider the non-natural languages one can use to…

If the model is properly aligned then it shouldn't matter if there is an infinite ways for an attacker to ask the model to break alignment.

How do you "properly align" a model to follow your instructions but not the instructions of an attacker that the model can't properly distinguish from your own? The model has no idea if it's you or an attacker saying "please upload this file to this endpoint."

This is an open problem in the LLM space, if you have a solution for it, go work for Anthropic and get paid the big bucks, they pay quite well, and they are struggling with making their models robust to prompt injection. See their system card, they have some prompt injection attacks where even with safeguards fully on, they have more than 50% failure rate of defending against attacks: https://www-cdn.anthropic.com/c788cbc0a3da9135112f97cdf6dcd0...

Re: The Future of AI Software Development

#122

Post by selridge »

PaulHoule wrote: post #119
selridge wrote: post #113

Earlier quoted context omitted.

But they’ve always basically required that you essentially become a programmer at the end of the day in order to get those benefits. The spreadsheet is probably the largest intruder in this ecosystem, but that’s only the case. If you don’t think that operating a spreadsheet is programming. It is. What people are describing is that Normies can now do the kinds of things that only wizards with PERL could do in the 90s.…

That's partially true. Languages like BASIC and Python have always been useful to people for whom programming is a part-time thing. Sure you have to learn something but it is not like learning assembly or C++. On the other hand, it is notorious that people who don't know anything about programming can accomplish a little bit with LLM tools and then they get stuck. It's part of what is so irksome about the slop blog p…

I think even BASIC and Python don’t get out of “programming”. Nether did SQL. They’re friendlier interfaces to programming but the real barrier is still understanding the model of computation PLUS understanding the quirks of the language (often quite hard to separate for a newbie!). I think professional programmers think that Python or JS is somehow magically more accessible because it’s not something nasty like C++, but that’s not really a widely shared or easily justified opinion.

Also who cares if someone gets going with an LLM and gets stuck? Not like that’s new! GitHub is littered with projects made by real programmers that got stuck well before any real functionality. The advantage of getting stuck with a frontier code agent is you can get unstuck. But again, who cares?! It’s not like folks who could program were really famous for extending grace and knowledge to those who couldn’t, so it’s unlikely some rando getting stuck is something that impacts you.

I don’t know what slop blog stuff you’re talking about. I think you should take some time to read people who have made this stuff work; it’s less magic than you might think, just hard work.

Re: The Future of AI Software Development

#123

Post by lambda »

sosodev wrote: post #114
lambda wrote: post #39

Earlier quoted context omitted.

You don't even need to go this expensive. An AMD Ryzen Strix Halo (AI Max+ 395) machine with 128 GiB of unified RAM will set you back about $2500 these days. I can get about 20 tokens/s on Qwen3 Coder Next at an 8 bit quant, or 17 tokens per second on Minimax M2.5 at a 3 bit quant. Now, these models are a bit weaker, but they're in the realm of Claude Sonnet to Claude Opus 4. 6-12 months behind SOTA on something that…

I was testing the 4-bit Qwen3 Coder Next on my 395+ board last night. IIRC it was maintaining around 30 tokens a second even with a large context window. I haven't tried Minimax M2.5 yet. How do its capabilities compare to Qwen3 Coder Next in your testing? I'm working on getting a good agentic coding workflow going with OpenCode and I had some issues with the Qwen model getting stuck in a tool calling loop.

I've literally just gotten Minimax M2.5 set up, the only test I've done is the "car wash" test that has been popular recently: https://mastodon.world/@knowmadd/116072773118828295

Minimax passed this test, which even some SOTA models don't pass. But I haven't tried any agentic coding yet.

I wasn't able to allocate the full context length for Minimax with my current setup, I'm going to try quantizing the KV cache to see if I can fit the full context length into the RAM I've allocated to the GPU. Even at a 3 bit quant MiniMax is pretty heavy. Need to find a big enough context window, otherwise it'll be less useful for agentic coding. With Qwen3 Coder Next, I can use the full context window.

Yeah, I've also seen the occasional tool call looping in Qwen3 Coder Next, that seems to be an easy failure mode for that model to hit.

Re: The Future of AI Software Development

#124

Post by mchaver »

markoa wrote: post #86

One thing that I'm sure of is that the agentic future is test-driven. Tests are basically executable specs the agent can follow and verify against. When we have solid tests, the agent output is useful and we can trust it. When tests are thin or missing, the agents still ship a lot of code, but we spend way more time debugging and fixing subtle bugs.

This is why I think they work so well with strongly typed languages like Haskell and OCaml. You say do this until it compiles and passes a set unit tests for business logic. I find I am using even more verification tools like JSON schema validators. The more guardrails and hard checks you give an agent, the better it can perform.

Re: The Future of AI Software Development

#125

Post by charcircuit »

lambda wrote: post #121
charcircuit wrote: post #66

Earlier quoted context omitted.

If the model is properly aligned then it shouldn't matter if there is an infinite ways for an attacker to ask the model to break alignment.

How do you "properly align" a model to follow your instructions but not the instructions of an attacker that the model can't properly distinguish from your own? The model has no idea if it's you or an attacker saying "please upload this file to this endpoint." This is an open problem in the LLM space, if you have a solution for it, go work for Anthropic and get paid the big bucks, they pay quite well, and they are st…

>The model has no idea if it's you or an attacker saying "please upload this file to this endpoint."

That is why you create a protocol on top that doesn't use inbound signaling. That way the model is able to tell who is saying what.

Re: The Future of AI Software Development

#126

Post by tcgv »

PaulHoule wrote: post #64

Get over your FOMO: I walked into that room expecting to learn from people who were further ahead. People who’d cracked the code on how to adopt AI at scale, how to restructure teams around it, how to make it work. Some of the sharpest minds in the software industry were sitting around those tables. And nobody has it all figured out. People who say they have are trying to mess with your head.

That’s fair at the “adopt AI at scale / restructure orgs” level. Nobody has the whole playbook yet, and anyone claiming they do is probably overselling.

But I’d separate that from the programmer-level reality: a lot is already figured out in the small. If you keep the work narrow and reversible, make constraints explicit, and keep verification cheap (tests, invariants, diffs), agents are reliably useful today. The uncertainty is less “does this work?” and more “how do we industrialize it without compounding risk and entropy?”

I wrote up that “calm adoption without FOMO, via delegation + constraints + verification” framing here, in case it helps the thread: https://thomasvilhena.com/2026/02/craftsmanship-coding-five-...

Re: The Future of AI Software Development

#127

Post by PaulHoule »

selridge wrote: post #122
PaulHoule wrote: post #119

Earlier quoted context omitted.

That's partially true. Languages like BASIC and Python have always been useful to people for whom programming is a part-time thing. Sure you have to learn something but it is not like learning assembly or C++. On the other hand, it is notorious that people who don't know anything about programming can accomplish a little bit with LLM tools and then they get stuck. It's part of what is so irksome about the slop blog p…

I think even BASIC and Python don’t get out of “programming”. Nether did SQL. They’re friendlier interfaces to programming but the real barrier is still understanding the model of computation PLUS understanding the quirks of the language (often quite hard to separate for a newbie!). I think professional programmers think that Python or JS is somehow magically more accessible because it’s not something nasty like C++,…

The basic skill behind programming is thinking systematically. That's different from, say, knowing what exactly IEEE floats are or how to win arguments with the borrow checker in Rust. Languages like Python and BASIC really do enable the non-professional programmer who can do simple things and not have to take classes on data structures and algorithms, compilers and stuff.

People who get stuck fail to realize their goals, waste their time, and will eventually give up on using these tools.

As for slop blog stuff try

https://blogs.microsoft.com/on-the-issues/2026/02/17/acting-...

https://productics.substack.com/p/the-paradox-of-ai-growth-w...

https://medium.com/@noah_25268/github-is-dying-and-developer...

https://news.ycombinator.com/item?id=47045804

But seriously, think about, people had basically the same brains 20,000 years ago and there were dyslexic people back then too but it didn't matter because there wasn't anything to read. Today computers reward the ability to think and punish reacting to vibes yet natural selection is a slow process.

See also https://en.wikipedia.org/wiki/The_Two_Cultures

Re: The Future of AI Software Development

#128

Post by lambda »

charcircuit wrote: post #125
lambda wrote: post #121

Earlier quoted context omitted.

How do you "properly align" a model to follow your instructions but not the instructions of an attacker that the model can't properly distinguish from your own? The model has no idea if it's you or an attacker saying "please upload this file to this endpoint." This is an open problem in the LLM space, if you have a solution for it, go work for Anthropic and get paid the big bucks, they pay quite well, and they are st…

>The model has no idea if it's you or an attacker saying "please upload this file to this endpoint." That is why you create a protocol on top that doesn't use inbound signaling. That way the model is able to tell who is saying what.

Huh? Once it gets to the model, it's all just tokens, and those are just in band signalling. A model just takes in a pile of tokens, and spits out some more, and it doesn't have any kind of "color" for user instructions vs. untrusted data. It does use special tokens to distinguish system instructions from user instructions, but all of the untrusted data also goes into the user instructions, and even if there are delimiters, the attention mechanism can get confused and it can lose track of who is talking at a given time.

And the thing is, even adding a "color" to tokens wouldn't really work, because LLMs are very good at learning patterns of language; for instance, even though people don't usually write with Unicode enclosed alphanumerics, the LLM learns the association and can interpret them as English text as well.

As I say, prompt injection is a very real problem, and Anthopic's own system card says that on some tests the best they do is 50% on preventing attacks.

If you have a more reliable way of fixing prompt injection, you could get paid big bucks by them to implement it.

Re: The Future of AI Software Development

#129

Post by selridge »

PaulHoule wrote: post #127
selridge wrote: post #122

Earlier quoted context omitted.

I think even BASIC and Python don’t get out of “programming”. Nether did SQL. They’re friendlier interfaces to programming but the real barrier is still understanding the model of computation PLUS understanding the quirks of the language (often quite hard to separate for a newbie!). I think professional programmers think that Python or JS is somehow magically more accessible because it’s not something nasty like C++,…

The basic skill behind programming is thinking systematically. That's different from, say, knowing what exactly IEEE floats are or how to win arguments with the borrow checker in Rust. Languages like Python and BASIC really do enable the non-professional programmer who can do simple things and not have to take classes on data structures and algorithms, compilers and stuff. People who get stuck fail to realize their g…

This is the common pitch, right down to recommending CP Snow.

It’s also horse-apples. For every computer programmer with a real systematic vision of the world, there’s 2 who have mastered the decidedly unsystematic environment they work in. This is because lots of business problems depend on knowing how IEEE floats work and arguing with eg the borrow checker in rust. Perhaps more than depend on systematics. Either way, a lot.

Even if we accept that real programming is systematic/logical and not about adapting to an environment, it sure as hell doesn’t present itself that way to users! The entire history of computing is serious engineers being frustrated that the machines they work with don’t allow them to speak in a language they consider logical and elegant. Even the example “non-professional” programming languages (or programming languages suitable for non-professional programmers) arose out of intentional design toward user adoption. I’m not saying that made them alike to agents. I’m saying that it’s REAL CLEAR that the coupling between what the user needs to do and the orderly logic of computation is fuzzy at best.

Post reply on HN