Oh another one,[1] cautious somewhat-skeptic edition.
A look at Cloudflare's AI-coded OAuth library
41–50 of 170 posts
Re: A look at Cloudflare's AI-coded OAuth library
#42Earlier quoted context omitted.
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
LLMs make learning new material easier than ever. I use them a lot and I am learning new things at an insane pace in different domains. The maximalists and skeptics both are confusing the debate by setting up this straw man that people will be delegating to LLMs blindly. The idea that someone clueless about OAuth should develop an OAuth lib with LLM support without learning a lot about the topic is... Just wrong. Don…
On the flip side, I wanted to see what common 8 layer PCB stackups were yesterday. ChatGPT wasn't giving me an answer that really made sense. After googling a bit, I realized almost all of the top results were AI generated, and also had very little in the way of real experience or advice.
It was extremely frustrating.
Re: A look at Cloudflare's AI-coded OAuth library
#43Earlier quoted context omitted.
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
Most important question on this entire topic. Fast forward 30 years and modern civilisation is entirely dependent on our AI’s. Will deep insight and innovation from a human perspective perhaps come to a stop?
No. Even with power tools, construction and joinery are physical work and require strength and skill.
What is new is that you'll need the wisdom to figure out when the tool can do the whole job, and where you need to intervene and supervise it closely.
So humans won't be doing any less thinking, rather they'll be putting their thinking to work in better ways.
Re: A look at Cloudflare's AI-coded OAuth library
#44An approach I don't see discussed here is having different agents using different models critique architecture and test coverage and author tests to vet the other model's work, including reviewing commits. Certainly no replacement for human in the loop but it will catch a lot of goofy "you said to only check in when all the tests pass so I disabled testing because I couldn't figure out how to fix the tests".
Re: A look at Cloudflare's AI-coded OAuth library
#45Earlier quoted context omitted.
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
This, for me, has been the question since the beginning. I’m yet to see anyone talk/think about the issue head on too. And whenever I’ve asked someone about it, they’ve not had any substantial thoughts.
Engineers will still exist and people will vibe code all kinds of things into existence. Some will break in spectacular ways, some of those projects will die, some will hire a real engineer to fix things.
I cannot see us living in a world of ignorance where there are literally zero engineers and no one on the planet understands what's been generated. Weirdly we could end up in a place where engineering skills are niche and extremely lucrative.
Re: A look at Cloudflare's AI-coded OAuth library
#46> At ForgeRock, we had hundreds of security bugs in our OAuth implementation, and that was despite having 100s of thousands of automated tests run on every commit, threat modelling, top-flight SAST/DAST, and extremely careful security review by experts. Wow. Anecdotally it's my understanding that OAuth is ... tricky ... but wow. Some would say it's a dumpster fire. I've never read the spec or implemented it.
Hundreds of thousands of tests? That sounds like quantity > quality or outright llm-generated ones, who even maintains them?
This was before LLMs. It was a combination of unit and end-to-end tests and tests written to comprehensively test every combination of parameters (eg test this security property holds for every single JWT algorithm we support etc). Also bear in mind that the product did a lot more than just OAuth.
Re: A look at Cloudflare's AI-coded OAuth library
#47Earlier quoted context omitted.
This, for me, has been the question since the beginning. I’m yet to see anyone talk/think about the issue head on too. And whenever I’ve asked someone about it, they’ve not had any substantial thoughts.
Engineers will still exist and people will vibe code all kinds of things into existence. Some will break in spectacular ways, some of those projects will die, some will hire a real engineer to fix things. I cannot see us living in a world of ignorance where there are literally zero engineers and no one on the planet understands what's been generated. Weirdly we could end up in a place where engineering skills are nic…
[deleted]
Re: A look at Cloudflare's AI-coded OAuth library
#48Earlier quoted context omitted.
I've been using an llm to do much of a k8s deployment for me. It's quick to get something working but I've had to constantly remind it to use secrets instead of committing credentials in clear text. A dangerous way to fail. I wonder if in my case this is caused by the training data having lots of examples from online tutorials that omit security concerns to focus on the basics.
> my case this is caused by the training data having I think it's caused by you not having a strong enough system prompt. Once you've built up a slightly reusable system prompt for coding or for infra work, where you bit by bit build it up while using a specific model (since different models respond differently to prompts), you end up getting better and better responses. So if you notice it putting plaintext credenti…
Yes, you are definitely right on that. I still find it a concerning failure mode. That said, maybe it's no worse than a junior copying from online examples without reading all the text some the code which of course has been very common also.
Re: A look at Cloudflare's AI-coded OAuth library
#49> What this interaction shows is how much knowledge you need to bring when you interact with an LLM. The “one big flaw” Claude produced in the middle would probably not have been spotted by someone less experienced with crypto code than this engineer obviously is. And likewise, many people would probably not have questioned the weird choice to move to PBKDF2 as a response For me this is the key takeaway. You gain pro…
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
In a few years hopefully the AI reviewers will be far more reliable than even the best human experts. This is generally how competency progresses in AI...
For example, at one point a human + computer would have been the strongest combo in chess, now you'd be insane to allow a human to critic a chess bot because they're so unlikely to add value, and statistically a human in the loop would be far more likely to introduce error. Similar things can be said in fields like machine vision, etc.
Software is about to become much higher quality and be written at much, much lower cost.
Re: A look at Cloudflare's AI-coded OAuth library
#50Part of me this "written by LLM" has been a way to get attention on the codebase and plenty of free reviews by domain expert skeptics, among the other goals (pushing AI efficiency to investors, experimenting, etc).