You found an interesting example of the fallacy of following and defining rules You can't define a system with only rules. Eventually, you run out of rules on how to apply the rules. Eventually, you need judgment and interpretation.
LLMs learn what programmers create, not how programmers work
21–30 of 31 posts
Re: LLMs learn what programmers create, not how programmers work
#22[flagged]
Re: LLMs learn what programmers create, not how programmers work
#23[dead]
Re: LLMs learn what programmers create, not how programmers work
#24And then they say AGI is coming
Re: LLMs learn what programmers create, not how programmers work
#25Not sure it is true LLMs don't see code or cli commands directly in their training. They go through reinforcement learning and they could easily be trained on a command line. People are paid to give human feedback. See https://huyenchip.com/2023/05/02/rlhf.html
Re: LLMs learn what programmers create, not how programmers work
#26I know even we hit the same thing building internal security tooling. our model kept formatting output like documentation, not like how we would or any person in place of us would read in a terminal at 2am during an incident. I am a bit curious, did you find this behavior consistent across models or is it more pronounced with certain ones?
I ran into it while building - I should have tested different temps too - I was just trying to get cli style tool calls to be more reliable
yeah temperature is probably worth a run, we noticed even small adjustments changed how the model interpreted formatting expectations quite a bit.
Re: LLMs learn what programmers create, not how programmers work
#27I know even we hit the same thing building internal security tooling. our model kept formatting output like documentation, not like how we would or any person in place of us would read in a terminal at 2am during an incident. I am a bit curious, did you find this behavior consistent across models or is it more pronounced with certain ones?
Literate programming is about to become mainstream in the funniest way possible.
oh yesss, except literate programming was still the human explaining intent to other humans. this is more like the human explaining intent to a machine that then explains it back to other humans. hahaha this is actually funny.
Re: LLMs learn what programmers create, not how programmers work
#28I ran into this building data pipelines for LLMs. Kept feeding structured JSON with numbers and the model would do bizarre arithmetic on them.
The moment I shifted to computing the analysis upstream and describing results in plain english, the outputs were much more coherent.
The model doesn't know what numbers/spreadsheets of data mean inherently, as in... the LLM inherently does not compute a math equation/formula within itself. Rather, an LLM would calls/creates calculation code on the side to then read off the results. And still with this side-work, the code/calculator created would give contextual wordings to the numbers; highest, average, x% growth, consolidating at y% rate, etc.
Re: LLMs learn what programmers create, not how programmers work
#29What changed after this discovery? What was better or worse and how better or worse?
Re: LLMs learn what programmers create, not how programmers work
#30Great insight. We noticed something similar when building our agent tool — LLMs would output git commit -m "..." with the quotes already formatted, even when we just wanted the message body. They're mimicking documentation examples, not actual command entry.
Ended up just accepting it and adjusting our parser. Fighting the model's training is a losing battle.