At their heart LLMs are basically a mechanism to guess(predict based on probability) the next word based on what it has already done/seen so far. How it goes about guessing them, or how it gets the context is based on
attention and
multihead attention functions. You could say that these functions provide which part(word) of a question/sentence must get a higher weight. That basically provides
context based on which it
predicts what needs to come next. How it predicts is basically your plain old neural network. Just like how in linear regression you know, if your model is predicting a straight line, you know in which area the next points are likely to appear. Similar mechanisms are used here to guess what the next words are likely to be.
It is a extreme auto complete feature in the context of code for sure. Note, LLMs are not sentient. That means they can't be held responsible for making decisions. Even more so code decisions.
Now you can argue its nothing special. But its some what like arguing eclipse/intellij are not special when compared to vim/emacs. That is just splitting hairs. IDE's definitely do a lot of productive work compared to plain text editors.
The initial demo's on LLMs confuse new users a lot. The demos go on the lines of giving a sentence like 'Implement a todo list app' or something like that and LLM writes some code implementing it. That's a wrong way to work with LLMs. Don't outsource your thinking or give it whole blanket problem statement to solve it. Think about LLMs like tools that do a lot of quick text writing for you you given the smallest, non ambiguous, atomically implementable/rollback statement possible.
It gets a while to get used to this, but once you are, you are more productive.