81–90 of 103 posts
Earlier quoted context omitted.
LLMs are the internal monologue.
They don't have N steps of iterative internal monologue before arriving at end result.
LLMs are the internal monologue, but they lack the ability to iterate - you have to pump them manually.
Earlier quoted context omitted.
Maybe it's just me, but I've yet to find a human who will confidently make up a function name that will do what I ask for and claim it is part of a standard library while that function does not exist at all. This is politician level stuff. It either means someone or something is trying to deceive in some way or doesn't have a complete understanding of the subject.
Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you…
Regarding my prior comment: The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. LLMs do not. At least not yet.
If the question is whether LLMs have an understanding of their output, then my argument just states that the fact that they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate.
As far as I understand the issue, this has nothing to do with the training. It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. Correct me if I am wrong here, I am not an expert on transformers. Just trying to wrap my head around this stuff.
Earlier quoted context omitted.
Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you…
To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actu…
My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious activity. My raw thought stream is about as good as LLM output - sounds right, but it's often full of nonsense. That's why I tend to say LLM output is akin to that unfiltered inner voice output.
My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things.
Also, fight-or-flight reaction may make you switch into full-on bullshitting mode, without it being intentional.
> they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate
And my argument is that this is the same thing a person's inner voice does. Deep understanding of a topic may make your inner voice output you more of the correct associations, but ultimately, most of the time, people have to have a back-and-forth with themselves in their own head to then say/write/do something correctly. With more complex (or less understood) tasks, this involves consciously performing an explicit step-by-step algorithm you track in your mind (and checklists in aviation and medicine are further externalization of that idea - to protect against you misremembering (or "hallucinating") the steps).
> It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output.
The output contains probability factors for predicted tokens, but chat interface doesn't expose it, and doesn't make a good use of it.
Earlier quoted context omitted.
To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actu…
> The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious acti…
There is a key difference between making up new facts and copying 'bullshit' picked up somewhere else. I agree that more than 50% of people don't put much effort into the accuracy of the material they recite, but that's not my point. My point is that LLMs obviously seem to make up completely new entities in generated code that do not have any merit in reality. I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Mine certainly does not. Does it generate non-sense that needs serious revision and filtering in general? Of course. But not in the style of new facts, just thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world.
Earlier quoted context omitted.
Yup. The whole premise of this test is idiotic. When you give a piece of code to an LLM, it doesn't magically switch into "code focused mode". They write: > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. But this is bullshit . Identifier renaming may be a no-op…
To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human programmer might well get tripped up by a name swap if they had missed the swap, or forgot it, but when presented with a short test of three lines like the one in the article's motivating example, the majority of human programmers would pass the test easily. T…
This is what I mean when I keep saying that a good analogy for current LLMs, GPT-4 in particular, is the "inner voice" - that bit in your head that spits out streams of thought using language. If you look at LLM output as you'd look at the thoughts popping into your head when you read the prompt, before you consciously fix/filter/"return to sender" them, I think you'd find the two very similar.
My objection to the article's test is thus that it's intentionally set up to confuse one's gut feel. "len" and "print" are not opaque tokens, they are words with whole lot of associations, and those associations are precisely why they were chosen to name those specific functions. Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level.
Since I claim that LLM output is equivalent to the thoughts produced by intuition, and there is no "conscious level" equivalent (unless you supply it yourself by a back-and-forth), I'd expect LLMs to have trouble with this test - and I see it disputing nothing but a bad mental model of the authors.
BTW. there is a way to test my claim, which is to construct similar tests explicitly designed to avoid going against training data and word-level associations (i.e. everything else the word used as function name, like "print" here, evokes). If GPT-4 fails that, where a typical person's "gut feel" wouldn't, then I'd see this as significant, and would be very interested in the exact details of the failure.
In fact, time allowing today/tomorrow, I'll try to do some tests like these myself.
Earlier quoted context omitted.
> The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious acti…
> My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or n…
Yes! Very often. That's because it's exactly "thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world".
In my actual code writing practice, I've spotted this manifesting in several ways. One, my subconscious often autocompletes some function name that it feels should be there (or would be nice to have it), which is immediately checked by the actual autocomplete of the IDE. Could be something like:
str.cou str.split Another case is when I write new code and I know functions don't exist - I just write what my stream of thoughts provide, and either clean it out later, or actually implement the functions that it "hallucinated".
Earlier quoted context omitted.
> My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or n…
> I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Yes! Very often. That's because it's exactly "thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world". In my actual code writing practice, I've spotted this manifesting in several ways. One, my…
> str.split > does not have a split() method.
Both confuse two existing entities from different languages/frameworks, which is exactly my point. These entities exist, they are not made up at all. They are just out of context. Ironically, the LLMs I used did rarely run into this kind of mistake (confusing similar entities of a different context).
As an example of what I got when asking a popular LLM for code that traverses the macOS accessibility UI element hierarchy: It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. Now I will admit that my mind is capable of doing the same. Under severe intoxication :)
On a side note: std::string should totally have a split() method and count() is more appropriate (as in intuitive) than size(). So I would say your brain is more "right" than the language in this case :)
Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject?
Earlier quoted context omitted.
Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you…
To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actu…
This has not been my experience.
Once you get out of the tech bubble you will frequently find people who genuinely treat talking as just pattern matching and signalling and speak words that don't correspond to any actual internal model of how the world works.
Earlier quoted context omitted.
To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human programmer might well get tripped up by a name swap if they had missed the swap, or forgot it, but when presented with a short test of three lines like the one in the article's motivating example, the majority of human programmers would pass the test easily. T…
My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. This is what I mean when I keep saying that a good analogy for current LLMs, GPT…
For the simple problem posed here, I dispute this. I do not need to rely on consciously overriding my intuition in order to replace "print" with "len" in this simple of an example. It's immediately obvious and, frankly, intuitive.
Every time somebody posts a negative result on LLMs' capabilities like that, somebody will go on HN, on Twitter, on Reddit, etc, and post an example where they get an LLM to produce a counter example of the negative result. Crucially, this follow-up "rebuttal" always seems to use a different prompt than the one in the original experiment. What can we conclude from that kind of sequence? We can conclude that neither t…
If you develop an LLM prompt that says "Add to 5" and prove that it works for a handful of cases, there's no guarantee it works for any other case. Whereas "(lambda (x) (+ x 5))" has some mechanical guarantee based on the implementation. And for the LLM, the negative result becomes more possible the higher the complexity of the prompt. This reduces LLMs to being suitable for tasks where reliability is not particularly important, or a supervisory person or system is checking for consistency and somehow can re-run the LLM to generate new results (which itself is not always reliable).