I tried implementing something like this over the summer but couldn't make progress with the 20-30 minimum response time for each OpenAI-generated block. From the demo video it looks like this runs pretty fast -- or does it?
Maccarone: AI-managed code blocks in Python
31–40 of 75 posts
Re: Maccarone: AI-managed code blocks in Python
#32There are so many languages with awesome type systems which can help guide AI to generate better code — and yet, these experiments always choose Python.
Python appears to be the best represented language in OpenAI's training set.
Re: Maccarone: AI-managed code blocks in Python
#33"Hallucination isn't a real problem, people will always scrutinize the generated code!" Sigh...
Where's this from? What are you quoting?
Re: Maccarone: AI-managed code blocks in Python
#34Earlier quoted context omitted.
OP has to be doing some parsing somewhere, so you just switch to seeking decorators rather than magic comments. It's still before the code reaches the interpreter. The potential issue I see here is that comments are valid anywhere while decorators may not be, but the parser is hopefully resilient to that. You could see a multi-phase LLM that uses the interpreter to ensure the code runs / works as expected
It literally just adds the code to the file between the comments. There’s nothing more magic than copy+paste and no runtime component. It’s completely different from a decorator that generates code at runtime ? As in, when the code runs ?
which comments should be generated?
what if the output has comments?
in my question, there is no need for the decorator to be handled at runtime, the tool that does the LLM stuff has to parse the Python code to know what to generate. It can just key off of decorators rather than comments, or at least this is my question & hypothesis.
Re: Maccarone: AI-managed code blocks in Python
#35This answer in the FAQ is wonderful: What prevents my program from behaving differently after each preprocessing run? - The strength of your faith in GPT-4.
Seems like we should bring back our old friend: the cache-invalidation key!
Re: Maccarone: AI-managed code blocks in Python
#36Earlier quoted context omitted.
Thorough code review is why we as an industry stopped shipping bugs.
We, as an industry, didn't stop shipping bugs. (Small example: https://github.com/CVEProject/cvelistV5/releases ) And that thorough code review prevents bugs is, at best, a debatable assertion. See e.g. https://www.microsoft.com/en-us/research/publication/code-re... It finds _some_ bugs. CI/CD, and a massive investment in automated testing has probably had the largest impact in moving software quality forward. (See e…
I'm fairly certain we ship far more bugs now than we ever did.
Before we had the ability to just add a patch and let the user download it, the end result needed to be very solid, because once that disk was purchased and taken home, it was static.
Now less attention is paid to these things, because it's just assumed to be tomorrow's problem.
Re: Maccarone: AI-managed code blocks in Python
#37This answer in the FAQ is wonderful: What prevents my program from behaving differently after each preprocessing run? - The strength of your faith in GPT-4.
[deleted]
Re: Maccarone: AI-managed code blocks in Python
#38"Hallucination isn't a real problem, people will always scrutinize the generated code!" Sigh...
[deleted]
Re: Maccarone: AI-managed code blocks in Python
#39I love it.
I'd like to make something more constrained. Instead of a fully-general programming language, let the LLM configure data-flows between pre-defined modules, field mappings, or presentations.
Then, hopefully, we could let the end-user more directly edit the prompt.
Re: Maccarone: AI-managed code blocks in Python
#40In theory, an AI that wrote proofs for their code (ala coq) could be used to validate preconditions specified by the developer, right?
I don't know to what extent you can self-verify your own system based on proof code you also write yourself. However I do know this topic of merging AI, Deep Learning, and proof assistants is an up and coming research area. I mainly follow publications from Talia Ringer [1] and collaborators on this topic.