Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

medium.com

Post reply on HN
Print view View original

101–110 of 154 posts

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#101

Post by ChadNauseam »

fer wrote: post #13
trjordan wrote: post #5

Seems like it could easily be training data set size as well. I'd love to see some quantification of errors in q/kdb+ (or hebrew) vs. languages of similar size that are left-to-right.

>Seems like it could easily be training data set size as well. I'm convinced that's the case. On any major LLM I can carpet bomb Java/Python boilerplate without issue. For Rust, at least last time I checked, it comes up with non-existing traits, more frequent hallucinations and general struggle to use the context effectively. In agent mode it turns into a first fight with the compiler, often ending in credit destroyi…

In my experience, claude works well at writing rust, and gemini is terrible. gemini writes rust as if it's a C++ programmer who has spent one day learning the basics of rust.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#102

Post by knome »

sitkack wrote: post #66
knome wrote: post #64

don't plan on it staying that way. I used to toss wads of my own forth-like language into LLMs to see what kinds of horrible failure modes the latest model would have in parsing and generating such code. at first they were hilariously bad, then just bad, then kind of okay, and now anthropic's claude4opus reads and writes it just fine.

How much incontext documentation for your language are you giving it, or does it just figure it out?

it varied. with the earlier models, generally more, trying to see if some apparition of mechanical understanding would eventually click into place.

IIRC, none of the gpt3 models did well with forth-like syntax. gpt4 generally did okay with it but could still get itself confused. claude4opus doesn't seem to have any trouble with it at all, and is happy to pick up the structures contextually, without explicit documentation of any sort.

another of my languages uses some parse transforming 'syntactic operators' that earlier models could never quite fully 'get', even with explanation. likely because at least one of them has no similar operator in popular languages. claude4opus, however, seems to infer them decently enough, and a single transform example is sufficient for it to generalize that understanding to the rest of the code it sees.

so far, claude has proved to be quite an impressive set of weights.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#103

Post by kstrauser »

threeducks wrote: post #97
kstrauser wrote: post #80

Earlier quoted context omitted.

I'd be surprised. That kind of thing was en vogue for a little while in the early 2000s before cooler heads prevailed, but now people will understandably shout at you for changing behavior in someone else's code. My guess is that nearly all packages that did this sort of thing were left behind in the 2-to-3 migration, which a lot of us used as the excuse for a clean break.

Not sure if that counts, but if you import both matplotlib and OpenCV at once, there is a good chance of a crash due to conflicting PyQt binaries: https://github.com/matplotlib/matplotlib/issues/29139 But I agree that observable side effects are generally pretty rare. And apparently, both libraries are not even in the top 100 packages, depending on how you count. It looks like those spots are all taken by libraries u…

Oof. That doesn't count in my opinion. The conflict is unfortunate, but it's not because either package is trying to modify other code. That is, the error is a side effect of how loading multiple libs into the interpret works. In theory, at least, you could fix those bugs without modifying the packages' behavior at all.

But still a bummer, to be sure. It's easy enough for me to say it doesn't count when I haven't been affected by it.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#106

Post by gowld »

short_sells_poo wrote: post #73
dotancohen wrote: post #22

Earlier quoted context omitted.

I encourage you to find some place that still uses a Hebrew typewriter. When they have to type numbers, they'll type the number in backwards. And an old Hebrew encoding also encoded characters in reverse order.

I think parent just means that "backwards" is a relative term. Your backwards is someone else's "forward". For someone who is used to reading Hebrew, they would be used to reading right to left and this would seem completely natural, no? Basically, the numbers 1234 and 4321 are identical assuming one is written left to right and the other is right to left. Then it's just a convention which way you are used to reading…

No, because Hebrew words are read right-to-left in Hebrew letters, but numbers are read left-to-right in Arabaic numerals. The direction of reading switches mid-sentence, but typewriters only type in one direction.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#107

Post by aidos »

semi-extrinsic wrote: post #92
0cf8612b2e1e wrote: post #60

Earlier quoted context omitted.

That does happen, but not frequently in the common libraries that are going to be in public training data. Is there a top 100 package that does something funny on import?

Dunno if it counts as funny, but the following code only works if you keep the matplotlib import: import matplotlib import PIL im = PIL.Image.open("foo.png")

What? There’s no way that’s correct. I use PIL exactly like that and don’t have matplotlib in my codebase.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#109

Post by FeepingCreature »

voxl wrote: post #89
cenamus wrote: post #50

Earlier quoted context omitted.

Is it hedging or did the training data just have lots of unecessary imports?

You already know the answer. Claude is not an intelligent sentient thing.

However, it possibly was RL trained on code tasks and penalized for errors.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#110

Post by saurik »

UltraSane wrote: post #56
clord wrote: post #31

There is something deep in this observation. When I reflect on how I write code, sometimes it’s backwards. Sometimes I start with the data and work back through to the outer functions, unnesting as I go. Sometimes I start with the final return and work back to the inputs. I notice sometimes LLMs should work this way, but can’t. So they end up rewriting from the start. Makes me wonder if future llms will be composing…

I think long term LLMs should directly generate Abstract Syntax Trees. But this is hard now because all the training data is text code.

The training data is text code that can be compiled, though, so the training data can also easily be an Abstract Syntax Tree.
Post reply on HN