There is something deep in this observation. When I reflect on how I write code, sometimes it’s backwards. Sometimes I start with the data and work back through to the outer functions, unnesting as I go. Sometimes I start with the final return and work back to the inputs. I notice sometimes LLMs should work this way, but can’t. So they end up rewriting from the start. Makes me wonder if future llms will be composing…
Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
111–120 of 154 posts
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#112Earlier quoted context omitted.
Dunno if it counts as funny, but the following code only works if you keep the matplotlib import: import matplotlib import PIL im = PIL.Image.open("foo.png")
What? There’s no way that’s correct. I use PIL exactly like that and don’t have matplotlib in my codebase.
import PIL.Image
Turns out that matplotlib (and probably lots of other stuff) does that, and then it gets resolved correctly.Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#113Earlier quoted context omitted.
What? There’s no way that’s correct. I use PIL exactly like that and don’t have matplotlib in my codebase.
Just try it. IIRC, to do the PIL import correctly you have to import PIL.Image Turns out that matplotlib (and probably lots of other stuff) does that, and then it gets resolved correctly.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#114There is something deep in this observation. When I reflect on how I write code, sometimes it’s backwards. Sometimes I start with the data and work back through to the outer functions, unnesting as I go. Sometimes I start with the final return and work back to the inputs. I notice sometimes LLMs should work this way, but can’t. So they end up rewriting from the start. Makes me wonder if future llms will be composing…
I think long term LLMs should directly generate Abstract Syntax Trees. But this is hard now because all the training data is text code.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#115Earlier quoted context omitted.
Tedious and annoying for one-off commands maybe. It's like regex. Pretty compelling if you're writing a one-off pattern, you get immediate feedback and then you throw it away. But it's not a good idea to use regexes in code that you're going to use long term. It's justifiable for simple regexes, and many people go against this advice, but really for anything remotely complex regexes become totally unreadable and extr…
What would you propose as an alternative to regexes that provides the same functionality without the unreadable syntax? I wrote something like that in C# once [0] but I'm not getting the impression that there's a lot of demand for that kind of thing. [0] https://github.com/Timwi/Generex
https://github.com/oils-for-unix/oils/wiki/Alternative-Regex...
I haven't actually used them because generally the right alternative to a regex is a proper parser.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#116Earlier quoted context omitted.
Yes. ChatGPT 4 and Claude 3.7. They led me to similar conclusions, but they produced very different syntax, which led me to believe that they were not just regurgitating from a common source.
Is there a reason you believe the models can accurately predict this sort of thing?
LLMs need all their context within easy reach. An LLM-first (for editing) language still has code comments and docstrings. Identifier names are long, and functions don't really need optional parameters. Strict typing is a must.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#117Earlier quoted context omitted.
> I did some experiments to find what a programming language would look like, instead of e.g. python, if it were designed to be written and edited by an LLM. Did your experiment consist of asking an LLM to design a programming language for itself?
Yes. ChatGPT 4 and Claude 3.7. They led me to similar conclusions, but they produced very different syntax, which led me to believe that they were not just regurgitating from a common source.
That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything
They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which you had no way of proving since you can't actually run that code) doesn't really mean anything
It would be similar to having it respond in a certain JSON format, they are great at that too. Doesn't really translate to a real world codebase
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#118Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#119Seems like it could easily be training data set size as well. I'd love to see some quantification of errors in q/kdb+ (or hebrew) vs. languages of similar size that are left-to-right.
W.r.t. natural languages, TFA clarifies it a bit:
> And it’s not the same as translation to Arabic or Hebrew; direction here refers to the temporal order in which the tokens are produced; even for right-to-left languages, the order in which the tokens get produced remains unchanged; rather, a thin display layer handles the visual presentation.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#120this thread made me revisit some past conversations with people like atw, nsl and aab with regard to possible ways to expose humans to the way rivers flow in k/q/apl land. the choices are limited, and decision takes some agony:
a) if you don't want your audience to close the tab right away, you'd say "a k expression is written, read and evaluated strictly right to left unless the precedence is explicitly overridden by parens, and this works better than you think, no worries, you'll come around. by the way, parens are evil, avoid them if you can".
b) if your intent is to retain a sharper crowd who went to yale or something, you'd say "a k expression is to be understood right of left", and throw them a freebie in form of a prompt for their local LLM in order to get lit. the magic sequence is just "f g h x leibniz".
for my own selfish reasons, i always chose the former, and it seems to perform better than the latter, proof:
https://github.com/kparc/ksimple
still, neither approach is anywhere near the chances of successfuly explaining which way to write python code to a 5yo kid, especially its precedence rules, which are much more intuitive (lol).
to explain the same thing to an LLM is not much different, really. all you need to do is to depress your 0yo kid with an obscene amount of _quality_ python code, of which there is no shortage. obviously, the more python code is fed to LLMs, the more humans will paste more LLM-generated python code, to be fed back to LLMs, ad lemniscate.
(and don't mind the future tense, we are already there)
============
so this is why LLMs can't write k/q/apl. first, they haven't seen enough of it. second, they are helpless to understand the meaning of a quote which was once chosen to helm a book known as SICP, not to mention countless human counterparts who came across it earlier, to the same effect:
"I think that it's extraordinarily important that we in computer science keep fun in computing. When it started out it was an awful lot of fun. Of course the paying customers got shafted every now and then and after a while we began to take their complaints seriously. We began to feel as if we really were responsible for the successful error-free perfect use of these machines. I don’t think we are. I think we're responsible for stretching them setting them off in new directions and keeping fun in the house. I hope the field of computer science never loses its sense of fun. Above all I hope we don’t become missionaries. Don't feel as if you're Bible salesmen. The world has too many of those already. What you know about computing other people will learn. Don’t feel as if the key to successful computing is only in your hands. What's in your hands I think and hope is intelligence: the ability to see the machine as more than when you were first led up to it that you can make it more."
― Alan J. Perlis