Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

medium.com

Post reply on HN
Print view View original

141–150 of 154 posts

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#141

Post by dotancohen »

cubefox wrote: post #136
gowld wrote: post #106

Earlier quoted context omitted.

No, because Hebrew words are read right-to-left in Hebrew letters, but numbers are read left-to-right in Arabaic numerals. The direction of reading switches mid-sentence, but typewriters only type in one direction.

Arguably Arabic numbers must always be read right-to-left, even in English, because the least significant digits can be read in order, while the value of the most significant digit depends on the number of less significant digits to the right. So in Hebrew the general reading direction actually fits Arabic numbers better.

And the Arabs actually say the numbers from right to left. It's "one and fifty", not "fifty one".

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#142

Post by cubefox »

dotancohen wrote: post #141
cubefox wrote: post #136

Earlier quoted context omitted.

Arguably Arabic numbers must always be read right-to-left, even in English, because the least significant digits can be read in order, while the value of the most significant digit depends on the number of less significant digits to the right. So in Hebrew the general reading direction actually fits Arabic numbers better.

And the Arabs actually say the numbers from right to left. It's "one and fifty", not "fifty one".

It's also written right to left. And in general, natural language is "little-endian": Less significant information tends to be mentioned first.[1]

1: https://www.thoughtco.com/given-before-new-principle-linguis...

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#143

Post by kenjackson »

astrange wrote: post #121
kenjackson wrote: post #114

Earlier quoted context omitted.

It's possible that LLMs build ASTs internally for programming. I have no 1st hand data on this, but it would not surprise me at all.

LLMs don't have memory, so they can't build anything. Insofar as they produce correct results, they have implicit structures corresponding to ASTs built into their networks during training time.

"LLMs don't have memory"

That's interesting. Is there research into adding memory or has it been proven that it provides no pragmatic value over any context it outputs?

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#145

Post by 0cf8612b2e1e »

refibrillator wrote: post #133
0cf8612b2e1e wrote: post #60

Earlier quoted context omitted.

That does happen, but not frequently in the common libraries that are going to be in public training data. Is there a top 100 package that does something funny on import?

Well “import torch” for example will resolve certain dynamically linked symbols, which must be done first before importing your own .so code that uses libtorch and pybind11. If not you will get a super fun to debug segfault, leaving you staring at gdb backtrace output while you ponder your career choice. This is buried deep in the PyTorch docs and I don’t have the willpower to go find it right now, sorry.

Heh. A ML library was my sneaking suspicion of where there might be something unexpected. Anything goes for performance and/or to get Nvidia to cooperate.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#146

Post by dotancohen »

weird-eye-issue wrote: post #135
dotancohen wrote: post #132

Earlier quoted context omitted.

> That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything The experiment was checking how well another unrelated LLM could write code using the syntax. And then in the reverse direction in new sessions. > They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which…

> And it influenced how I use LLMs for work, for the better How so?

I let the LLM come up with all the boiler plate classes, functions, modules, etc that it wants. I let it name things. I let it design the API. But what I don't let it do, is design the flow of operations. I come up with a flow chart as a flow of operations, and explain that to the LLM. Almost any if statement is a result of something I specifically mentioned.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#147

Post by kazinator »

School-Cotton wrote: post #83
catfacts wrote: post #82

Cognitive load in LLMs: When LLMs are faced with syntactic complexity (Lisp/J parentheses/RL-NOP), distractors (cat facts), or unfamiliar paradigms (right-to-left evaluation), the model’s performance degrades because its "attention bandwidth" is split or overwhelmed. This mirrors human cognitive overload. My question: is there a way to reduce cognitive load in LLMs?, one solution seems to be process the input and out…

Isn't the whole idea of Lisp that there is _no_ syntactic complexity? Lisp programs are roughly a serialized AST.

The idea in Lisp is that there is low complexity in encoding abstract syntax into surface syntax.

There can be considerable complexity in Lisp abstract syntax.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#148

Post by kazinator »

pornel wrote: post #95
School-Cotton wrote: post #83

Earlier quoted context omitted.

Isn't the whole idea of Lisp that there is _no_ syntactic complexity? Lisp programs are roughly a serialized AST.

LLMs use tokens, with 1d positions and rich complex fuzzy meanings, as their native "syntax", so for them LISP is alien and hard to process. That's like reading binary for humans. 1s and 0s may be the simplest possible representation of information, but not the one your wet neural network recognizes.

Already over two years ago, using GPT4, I experimented with code generation using a relatively unknown dialect of Lisp for which there are few online materials or discussions. Yet, the results were good. The LLM slightly hallucinated between that dialect and Scheme and Common Lisp, but corrected itself when instructed clearly. When given a verbal description of a macro that is available in the dialect, it was able to refactor the code to take advantage of it.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#149

Post by School-Cotton »

kazinator wrote: post #147
School-Cotton wrote: post #83

Earlier quoted context omitted.

Isn't the whole idea of Lisp that there is _no_ syntactic complexity? Lisp programs are roughly a serialized AST.

The idea in Lisp is that there is low complexity in encoding abstract syntax into surface syntax. There can be considerable complexity in Lisp abstract syntax.

Fair enough.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#150

Post by leprechaun1066 »

gabiteodoru wrote: post #127
leprechaun1066 wrote: post #48

It's not because of the left of right evaluation. If the difference was that simple, most humans, let alone LLMs, wouldn't struggle with picking up q when they come from the common languages. Usually when someone solves problems with q, they don't use the way one would for Python/Java/C/C++/C#/etc. This is probably a poor example, if I asked someone to write a function to create an nxn identity matrix for a given num…

Or, even better, also from the cookbook: {(2#x)#1,x#0} But this really borders on obfuscation :P

That does require someone to know that the take operator continues to treat the y list as circular when x is a list.

I think this form might be a bit easier: {(x,x)#(x*x)#1,x#0}

Post reply on HN