Writing Lisp is AI resistant and I'm sad
blog.djhaskin.com
Writing Lisp is AI resistant and I'm sad
1–10 of 104 posts
Re: Writing Lisp is AI resistant and I'm sad
#2> I wonder what adaptations will be necessary to make AIs work better on Lisp.
Some are going to nitpick that Clojure isn't as lispy as, say, Common Lisp but I did experiment with Claude Code CLI and my paid Anthropic subscription (Sonnet 4.6 mostly) and Clojure.
It is okay'ish. I got it to write a topological sort and pure (no side effect) functions taking in and returning non-totally-trivial data structures (maps in maps with sets and counters etc.). But apparently it's got problems with...
... drumroll ...
The number of parentheses. It's so bad that the author of figwheel (a successful ClojureScript project) is working on a Clojure MCP that fixes parens in Clojure code spoutted by AI (well the project does more than that, but the description literally says it's "designed to handle Clojure parentheses reliably").
You can't make that up: there's literally an issue with the number of closing parens.
Now... I don't think giving an AI access to a Lisp REPL and telling it: "Do this by bumping on the guardrails left and right until something is working" is the way to go (yet?) for Clojure code.
I'm passing it a codebase (not too big, so no context size issue) and I know what I want: I tell it "Write a function which takes this data structure in and that other parameter, the function must do xxx, the function must return the same data structure out". Before that I told it to also implement tests (relatively easy for they're pure functions) for each function it writes and to run tests after each function it implements or modify.
And it's doing okay.
Re: Writing Lisp is AI resistant and I'm sad
#3> I wonder what adaptations will be necessary to make AIs work better on Lisp. Some are going to nitpick that Clojure isn't as lispy as, say, Common Lisp but I did experiment with Claude Code CLI and my paid Anthropic subscription (Sonnet 4.6 mostly) and Clojure. It is okay'ish. I got it to write a topological sort and pure (no side effect) functions taking in and returning non-totally-trivial data structures (maps i…
Sometimes LLMs astonish me with what the code they can write. Other times I have to laugh or cry.
As an example, I asked claude 3.5 back when that was the latest to indent all the code in my file by four more spaces. The file was about 700 lines long. I got a busy spinner for two minutes then it said, "OK, first 50 lines done, now I'll do the rest" and got another busy spinner and it said, "this is taking too long. I'm going to write a program to do it", which of course it had no problem doing. The point is that it is superhuman at some things and completely brain-dead about others, and counting parens is one of those things I wouldn't expect it to be good at.
Re: Writing Lisp is AI resistant and I'm sad
#4I think some kind of graph-capable model directly on the AST or a lower level IR would be the way to go, with bidirectionality so that changes propagate back up to the syntax without squandering LLM resources.
Re: Writing Lisp is AI resistant and I'm sad
#5Interesting, and not quite my experience. While I do get better agentic coding results for Python projects, I also get good results working with Common Lisp projects. I do have a habit of opening an Emacs buffer and writing a huge prompt with documentation details, sometimes sample code in other languages or if I am hitting APIs I add a working CURL example. For Common Lisp my initial prompts are often huge, but I find thinking about a problem and prompt creation to be fun.
The article mentions a REPL skill. I don’t do that: letting model+tools run sbcl is sufficient.
Re: Writing Lisp is AI resistant and I'm sad
#6> I wonder what adaptations will be necessary to make AIs work better on Lisp. Some are going to nitpick that Clojure isn't as lispy as, say, Common Lisp but I did experiment with Claude Code CLI and my paid Anthropic subscription (Sonnet 4.6 mostly) and Clojure. It is okay'ish. I got it to write a topological sort and pure (no side effect) functions taking in and returning non-totally-trivial data structures (maps i…
Sometimes LLMs astonish me with what the code they can write. Other times I have to laugh or cry. As an example, I asked claude 3.5 back when that was the latest to indent all the code in my file by four more spaces. The file was about 700 lines long. I got a busy spinner for two minutes then it said, "OK, first 50 lines done, now I'll do the rest" and got another busy spinner and it said, "this is taking too long. I…
I think LLMs are great at compression and information retrieval, but poor at reasoning. They seem to work well with popular languages like Python because they have been trained with a massive amount of real code. As demonstrated by several publications, on niche languages their performance is quite variable.
Re: Writing Lisp is AI resistant and I'm sad
#7Wildly speculating here, but if you buy that human brains have innate / evolved syntactic knowledge, and that this knowledge projects itself as the common syntactic forms across the bulk of human languages, then it’s no surprise that LLMs don’t have particularly deep grooves for s-expressions, regardless of the programming language distribution of the training set.
Re: Writing Lisp is AI resistant and I'm sad
#8I am a bit (ok very) worried AI will most likely kill language diversity in programming. I also don't see it settling on a more optimal solution it will probably just use the most available languages out there and be very hard to push out of that rut. And it's not limited to languages I expect knowledge ruts all over the place and due to humans and AI choosing the path of least resistance I don't see an active way to fight this.
Re: Writing Lisp is AI resistant and I'm sad
#9Be careful what you wish for
Re: Writing Lisp is AI resistant and I'm sad
#10I've had it write Scheme with little issue -- it even completely the latter half of a small toy compiler. I think the REPL is the issue, not the coding; forcing it to treat the REPL like another conversation participant is likely the only way for that to work, and this article does not handle it that way. Instead, hand it a compiler and let it use the workflow it is optimized for.