Live data from Hacker News

Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

medium.com

131–140 of 154 posts

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#131
post #55

Earlier quoted context omitted.

With enough resolution and appropriately chosen transformation steps, it is equivalent. E.g., the diffusion could focus on one region and then later focus on another, and it's allowed to undo the effort it did in one region. Nothing architecturally prohibits that solution style from emerging.

The choice of transformation steps to facilitate this specific diffuse approach seems like a non-trivial problem. It doesn't follow such an organic solution would emerge at all, now, does it?

The pattern ", now, " is indicative of a sort of patronization I don't normally engage with, but, yes, you're correct.

In some measure of agreeing with you: For other classes of models we know for a fact that there exist problems which can be solved by those architectures and which can't be trained using current techniques. It doesn't feel like a huge stretch that such training-resistent data might exist for diffusion models.

That said, I still see three problems. Notably, the current ancestral chain of inquiry seems to care about the model and not the training process, so the point is moot. Secondarily, in other similar domains (like soft circuits) those organic solutions do seem to emerge, suggesting (but not proving) that the training process _is_ up to par. Lastly, in other related domains, when such a solution doesn't emerge it ordinarily happens because some simpler methodology achieves better results, meaning that even with individual data points suggesting that diffusion solutions don't model that sort of linearity you still need to work a little bit to prove that such an observation actually matters.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#132

Earlier quoted context omitted.

Yes. ChatGPT 4 and Claude 3.7. They led me to similar conclusions, but they produced very different syntax, which led me to believe that they were not just regurgitating from a common source.

Great so your experiment just consisted of having an LLM hallucinate That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which you had no way of proving since you can't actually run that code) doesn't reall…

  > That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything
The experiment was checking how well another unrelated LLM could write code using the syntax. And then in the reverse direction in new sessions.

  > They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which you had no way of proving since you can't actually run that code) doesn't really mean anything
Of course I could check the code. I had no compiler for it, but "running" code in one's head without a compiler is something first year students get very good at in their Introduction To C course. And checking how they edit and modify the code.

This isn't a published study, it was an experiment. And it influenced how I use LLMs for work, for the better. I'd even call that a successful experiment, now that I better understand the strengths and limitations of LLMs in this field.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#133

Earlier quoted context omitted.

Especially in Python, where it can be hard to tell if something is being imported purely for side effects.

That does happen, but not frequently in the common libraries that are going to be in public training data. Is there a top 100 package that does something funny on import?

Well “import torch” for example will resolve certain dynamically linked symbols, which must be done first before importing your own .so code that uses libtorch and pybind11. If not you will get a super fun to debug segfault, leaving you staring at gdb backtrace output while you ponder your career choice.

This is buried deep in the PyTorch docs and I don’t have the willpower to go find it right now, sorry.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#134
post #23

Earlier quoted context omitted.

I don't know what counts as a major model. Relevant to this, I've dabbled with Gemma, Qwen, Mistral, Llama, Granite and Phi models, mostly 3-14b varieties but also some larger ones on CPU on a machine that has 64 GB RAM.

I think the issue there is those smaller versions of those models. I regularly use Gemma3 and Qwen3 for programming without issue but in the 27b-32b range. Going smaller than that generally yields garbage.

I've tried 24-32b sizes as well and besides being even slower they were also unreliable.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#135

Earlier quoted context omitted.

Great so your experiment just consisted of having an LLM hallucinate That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which you had no way of proving since you can't actually run that code) doesn't reall…

> That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything The experiment was checking how well another unrelated LLM could write code using the syntax. And then in the reverse direction in new sessions. > They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which…

> And it influenced how I use LLMs for work, for the better

How so?

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#136
post #106

Earlier quoted context omitted.

I think parent just means that "backwards" is a relative term. Your backwards is someone else's "forward". For someone who is used to reading Hebrew, they would be used to reading right to left and this would seem completely natural, no? Basically, the numbers 1234 and 4321 are identical assuming one is written left to right and the other is right to left. Then it's just a convention which way you are used to reading…

No, because Hebrew words are read right-to-left in Hebrew letters, but numbers are read left-to-right in Arabaic numerals. The direction of reading switches mid-sentence, but typewriters only type in one direction.

Arguably Arabic numbers must always be read right-to-left, even in English, because the least significant digits can be read in order, while the value of the most significant digit depends on the number of less significant digits to the right. So in Hebrew the general reading direction actually fits Arabic numbers better.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#137
post #110

Earlier quoted context omitted.

I think long term LLMs should directly generate Abstract Syntax Trees. But this is hard now because all the training data is text code.

The training data is text code that can be compiled, though, so the training data can also easily be an Abstract Syntax Tree.

But is anyone actually doing this?

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#139
post #102
post #66

Earlier quoted context omitted.

How much incontext documentation for your language are you giving it, or does it just figure it out?

it varied. with the earlier models, generally more, trying to see if some apparition of mechanical understanding would eventually click into place. IIRC, none of the gpt3 models did well with forth-like syntax. gpt4 generally did okay with it but could still get itself confused. claude4opus doesn't seem to have any trouble with it at all, and is happy to pick up the structures contextually, without explicit documenta…

That is excellent, I am also using it to prototype designing languages and 3.7 and 4.0 models are really quite good for this. I haven't found substantial academic research in using LLMs for making prototype language compilers.

Re: Why LLMs Can't Write Q/Kdb+: Writing Code Right-to-Left

#140
post #125

How do they do with lisps?

(en passant, k is arguably more lispy than some lisps. for a lisp guy, the first cultural shock is usually the absence of 99% of superfluous parens) as for LLM copilots and the quality of their lisp: why you'd expect them to excel in lisp better than they lisp in excel, pardon the pun?

I was curious because there is a much larger training corpus for Lisps, so if the problem really is one of training data rather ordering then this would be a way of showing that.
Post reply on HN