As a non developer, this was incredible useful to understand the prompt structure and apply to my own Claude Code.
A look at Cloudflare's AI-coded OAuth library
161–170 of 170 posts
Re: A look at Cloudflare's AI-coded OAuth library
#162Earlier quoted context omitted.
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
LLMs make learning new material easier than ever. I use them a lot and I am learning new things at an insane pace in different domains. The maximalists and skeptics both are confusing the debate by setting up this straw man that people will be delegating to LLMs blindly. The idea that someone clueless about OAuth should develop an OAuth lib with LLM support without learning a lot about the topic is... Just wrong. Don…
> LLMs make learning new material easier than ever. I use them a lot and I am learning new things at an insane pace in different domains.
Sorry, but the the amount of bad information dispensed by models and the student's ability to go "hey, that's wrong" due to a lack of experience and knowledge means that this is going to lead to disaster very often.
People already dispensing terrible information on YouTube because they trusted an AI to generate their voice-over script to explain something when creating learning materials.
Re: A look at Cloudflare's AI-coded OAuth library
#163Earlier quoted context omitted.
In this case it would need to use its own partitioning algorithm because of some specific ordering guarantees we care about.
Then rewrite them to another topic. Nevermind, complex multithreading sounds like the better solution
There’s more to it than that. We don’t care about total order even within partitions. Every so often we get a message that must not be sent downstream until some subset of messages have been sent.
So most of the time we’re fine sending 100-200 parallel message batches, but sometimes we need to stop and wait for some batches to complete before sending any more.
We also want to control how hard we hammer specific resources downstream, which don’t correlate with the partitions we’d need. Additionally we want to scale up and scale down the parallelism per each of the previously mentioned resources depending on how fast they are coming in to maximize batch size (while keeping latency low).
There’s of course ways to do this with multiple partitions by having the consumers communicate with each other. But now we have added an additional consumer and topic to the pipeline, and an inter-consumer control system.
It was overall easier to have one consumer read from the existing topic and spawn goroutines, so that we can have more dynamic control, the ability to scale up and down immediately without worrying about rebalancing, and easy communication between threads.
Re: A look at Cloudflare's AI-coded OAuth library
#164> What this interaction shows is how much knowledge you need to bring when you interact with an LLM. The “one big flaw” Claude produced in the middle would probably not have been spotted by someone less experienced with crypto code than this engineer obviously is. And likewise, many people would probably not have questioned the weird choice to move to PBKDF2 as a response For me this is the key takeaway. You gain pro…
I've found llms are very quick to add defaults, fallbacks, rescues–which all makes it very easy for code to look like it is working when it is not or will not. I call this out three different places in my CLAUDE.md trying to adjust for this, and still occasionally get.
Agreed, I’ve found that LLMs favor wrapping entire functions in broad try:catch structures, making it nearly impossible to identify and debug problems. Dangerous if you don’t correct it.
Re: A look at Cloudflare's AI-coded OAuth library
#165> What this interaction shows is how much knowledge you need to bring when you interact with an LLM. The “one big flaw” Claude produced in the middle would probably not have been spotted by someone less experienced with crypto code than this engineer obviously is. And likewise, many people would probably not have questioned the weird choice to move to PBKDF2 as a response For me this is the key takeaway. You gain pro…
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
Same place the previous generation came from - college courses, DIY hackers, and learn-on-the-job training and apprenticeships.
Why would that change?
Re: A look at Cloudflare's AI-coded OAuth library
#166Earlier quoted context omitted.
In a few years hopefully the AI reviewers will be far more reliable than even the best human experts. This is generally how competency progresses in AI... For example, at one point a human + computer would have been the strongest combo in chess, now you'd be insane to allow a human to critic a chess bot because they're so unlikely to add value, and statistically a human in the loop would be far more likely to introdu…
My prediction is that for that to happen we’ll need to figure out a way to measure software quality in the way we can measure a chess game, so that we can use synthetic data to continue improving the models. I don’t think we are anywhere close to doing that.
Agreed, chess is for all intents and purposes a ‘solved’ game, it’s merely a matter of processing power, and even then we have shortcuts that are easily ‘good enough’ to play perfectly against human opponents.
But how do you reduce the requirements for software to something so simple and elegant as chess rules? Is it foolish to assume that if we could have, we already would have? Even for humans, the process of writing software includes a lot of guess-and-check most of the time - the idea that you could sit down and think through every aspect of software, then describe it immaculately, then translate that description to a working solution with no bugs or review or need for course correction is just… it’s a pipe dream.
Re: A look at Cloudflare's AI-coded OAuth library
#167Earlier quoted context omitted.
Learning from LLMs is akin to learning from Joe Rogan. You are getting a stylised view of a topic from an entity who lacks the deep understanding needed to be able to fully distill the information. But it is enough to gain enough knowledge for you to feel confident which is still valuable but also dangerous. And I assure you that many, many people are delegating to LLMs blindly e.g. it's a huge problem in the UK lega…
> You are getting a stylised view of a topic from an entity who lacks the deep understanding Isn't this how every child learns? Unless his father happens to be king of Macedonia, of course.
Learning requirements of a child are different from an adult.
They are fine with a lossy, imperfect version of how the world works.
Re: A look at Cloudflare's AI-coded OAuth library
#168Earlier quoted context omitted.
I've been using an llm to do much of a k8s deployment for me. It's quick to get something working but I've had to constantly remind it to use secrets instead of committing credentials in clear text. A dangerous way to fail. I wonder if in my case this is caused by the training data having lots of examples from online tutorials that omit security concerns to focus on the basics.
> my case this is caused by the training data having I think it's caused by you not having a strong enough system prompt. Once you've built up a slightly reusable system prompt for coding or for infra work, where you bit by bit build it up while using a specific model (since different models respond differently to prompts), you end up getting better and better responses. So if you notice it putting plaintext credenti…
Seems like this is the sort of thing the tooling should do for you. Maybe have a few preset prompts for different contexts. I shouldn't have to type in a magic phrase to get it to write code that follows basic professional practice.
Re: A look at Cloudflare's AI-coded OAuth library
#169Earlier quoted context omitted.
> my case this is caused by the training data having I think it's caused by you not having a strong enough system prompt. Once you've built up a slightly reusable system prompt for coding or for infra work, where you bit by bit build it up while using a specific model (since different models respond differently to prompts), you end up getting better and better responses. So if you notice it putting plaintext credenti…
Seems like this is the sort of thing the tooling should do for you. Maybe have a few preset prompts for different contexts. I shouldn't have to type in a magic phrase to get it to write code that follows basic professional practice.
> type in a magic phrase to get it to write code that follows basic professional practice
The thing is, there is no agreed upon "basic professional practice" that could be encoded. Ask 10 programmers what "basic professional practice" is and you're gonna get 10 different answers.
For me, simplicity, avoiding over-engineering and being deliberately slow are the best ways to program. But lots of people disagree with this, especially the last part, so why have a different model for every preference, when you can use system prompts to have one model that could follow all of it?
Re: A look at Cloudflare's AI-coded OAuth library
#170Earlier quoted context omitted.
I agree that most code reviewers are pretty bad at spotting subtle bugs in code that looks good superficially. I have a lot of experience reviewing code -- more than I ever really wanted. It has... turned me cynical and bitter, to the point that I never believe anything is right, no matter who wrote it or how nice it looks, because I've seen so many ways things can go wrong. So I tend to review every line, simulate i…
I agree with Kenton’s aside.
> I agree with Kenton...
https://news.ycombinator.com/user?id=eastdakota
> CEO & co-founder of CloudFlare
No kidding, huh?