Ordering issues can be overcome by allowing the model to think in one direction and then reverse the output once it has created it.
Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
71–80 of 154 posts
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#72There is something deep in this observation. When I reflect on how I write code, sometimes it’s backwards. Sometimes I start with the data and work back through to the outer functions, unnesting as I go. Sometimes I start with the final return and work back to the inputs. I notice sometimes LLMs should work this way, but can’t. So they end up rewriting from the start. Makes me wonder if future llms will be composing…
> Sometimes I start with the final return and work back to the inputs. Shouldn't be hard to train a coding LLM to do this too by doubling the training time: train the LLM both forwards and backwards across the training data.
GP is talking about the nonlinear way that software engineers think, reason, and write down code. Simply doing the same thing but backwards provides no benefit.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#73Earlier quoted context omitted.
No no, the second character you write must always be temporally preceded by the character you wrote first. Otherwise the second wouldn't have been the second, but the first, and moreover, the first would have been the second, which it wasn't.
I encourage you to find some place that still uses a Hebrew typewriter. When they have to type numbers, they'll type the number in backwards. And an old Hebrew encoding also encoded characters in reverse order.
I think parent just means that "backwards" is a relative term. Your backwards is someone else's "forward". For someone who is used to reading Hebrew, they would be used to reading right to left and this would seem completely natural, no?
Basically, the numbers 1234 and 4321 are identical assuming one is written left to right and the other is right to left. Then it's just a convention which way you are used to reading.
I know nothing of Old (or New) Hebrew unfortunately so I may be completely off base.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#74Languages that are difficult for LLM to read & write are also difficult for the general public. These languages have always had poor uptake and never reach critical mass, or are eventually replaced by better languages. Language designers would be smart to recognize this fact and favor making their languages more LLM friendly. This should also make them more human friendly.
I don't find this to be true. There are languages that are difficult to wrap your head around initially, but that turn out to be delightfully productive with a few adjustments to the mental model. Adjustments that LLMs don't have the training data for.
That says nothing about the language at all, actually. Just that it's small and easily confused for something more idiomatic to a newbie.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#75Earlier quoted context omitted.
It might be a question of familiarity rather than objective usability. I'm writing this comment in Latin letters rather than Cyrillic or Hebrew because I find Latin letters much more usable than Cyrillic or Hebrew. But that's because I've been surrounded by Latin letters since I was born, and have only occasionally encountered Cyrillic or Hebrew. I think it's obvious that Cyrillic isn't any less usable than the Latin…
> " I suspect that the APL syntax ... is less usable for left-to-right readers" On the contrary, I find it much more usable for left-to-right readers, because it allows a "top-down" reading of the expressions, instead of a "bottom-up" reading. When trying to understand an unfamiliar program, for debugging or maintenance, you normally do not want to waste time by reading completely all expressions, which provide irrel…
Hmm, could be. I'll have to think about that.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#76It's not because of the left of right evaluation. If the difference was that simple, most humans, let alone LLMs, wouldn't struggle with picking up q when they come from the common languages. Usually when someone solves problems with q, they don't use the way one would for Python/Java/C/C++/C#/etc. This is probably a poor example, if I asked someone to write a function to create an nxn identity matrix for a given num…
A human can deal with right-to-left evaluation by moving the cursor around to write in that direction. An LLM can’t do that on its own. A human given an editor that can only append would struggle too.
Idea: feed the language model the parse tree instead of the textual sequence.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#77I always thought APL was written in the wrong direction. It writes like a concatenative language that's backwards--you tack things onto the front. NumPy fixes it by making the verbs all dotted function calls, effectively mirroring the order. e.g. in APL you write "10 10 ⍴ ⍳100" but in NumPy you write "np.arange(1, 101).reshape(10, 10)". Even if you don't know either language, you can tell that the APL version is the…
APL was designed as a notation for math; if you pronounce it properly, it makes more sense than numpy: The 10 by 10 reshaping of counting to 100
Numpy: Counting to 100, then reshaped to 10 x 10. Doesn't really seem all that different to me.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#78There is something deep in this observation. When I reflect on how I write code, sometimes it’s backwards. Sometimes I start with the data and work back through to the outer functions, unnesting as I go. Sometimes I start with the final return and work back to the inputs. I notice sometimes LLMs should work this way, but can’t. So they end up rewriting from the start. Makes me wonder if future llms will be composing…
I think long term LLMs should directly generate Abstract Syntax Trees. But this is hard now because all the training data is text code.
There's a fair amount of experimental work happening trying different parsing and resolution procedures such that the training data reflects an AST and or predicts nodes in an AST as an in-filling capability.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#79Seems like it could easily be training data set size as well. I'd love to see some quantification of errors in q/kdb+ (or hebrew) vs. languages of similar size that are left-to-right.
That’s what I thought. Lack of training data might be a reason.
Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left
#80Earlier quoted context omitted.
Especially in Python, where it can be hard to tell if something is being imported purely for side effects.
That does happen, but not frequently in the common libraries that are going to be in public training data. Is there a top 100 package that does something funny on import?
I'd be surprised. That kind of thing was en vogue for a little while in the early 2000s before cooler heads prevailed, but now people will understandably shout at you for changing behavior in someone else's code.
My guess is that nearly all packages that did this sort of thing were left behind in the 2-to-3 migration, which a lot of us used as the excuse for a clean break.