Recent AI model progress feels mostly like bullshit

Post by [deleted] » Sun, Apr 06, 2025, 11:52 PM UTC

MegaButts wrote: post #135

fancyfredbot wrote: post #129

Earlier quoted context omitted.

It didn't understand you but the response was plausible enough to require fact checking. Although that isn't literally indistinguishable from 'understanding' (because your fact checking easily discerned that) it suggests that at a surface level it did appear to understand your question and knew what a plausible answer might look like. This is not necessarily useful but it's quite impressive.

There are times it just generates complete nonsense that has nothing to do with what I said, but it's certainly not most of the time. I do not know how often, but I'd say it's definitely under 10% and almost certainly under 5% that the above happens. Sure, LLMs are incredibly impressive from a technical standpoint. But they're so fucking stupid I hate using them. > This is not necessarily useful but it's quite impres…

Post by Nesco » Mon, Apr 07, 2025, 12:01 AM UTC

Post by flutas » Mon, Apr 07, 2025, 12:08 AM UTC

arkmm wrote: post #31

flutas wrote: post #20

Earlier quoted context omitted.

FWIW 2.5-exp was the only one that managed to get a problem I asked it right, compared to Claude 3.7 and o1 (or any of the other free models in Cursor). It was reverse engineering ~550MB of Hermes bytecode from a react native app, with each function split into a separate file for grep-ability and LLM compatibility. The others would all start off right then quickly default to just greping randomly what they expected i…

How did you fit 550MB of bytecode into the context window? Was this using 2.5 in an agentic framework? (i.e. repeated model calls and tool usage)

    awk '/^=> \[Function #/ {           
        if (out) close(out);
        fn = $0; sub(/^.*#/, "", fn); sub(/ .*/, "", fn);
        out = "function_" fn ".txt"
    }
    { if (out) print > out }' bundle.hasm

Post by jcims » Mon, Apr 07, 2025, 12:11 AM UTC

aerhardt wrote: post #34

My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question: "Is Paul Newman known for having had problems with alcohol?" All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response: "Paul Newman is not widely known for having had problems with alcohol. While he portrayed charact…

    Paul Newman, the renowned actor and humanitarian, did not have a widely publicized 
    struggle with alcohol addiction throughout most of his life, but there were 
    specific instances that indicated challenges.

Post by goldfeld » Mon, Apr 07, 2025, 12:13 AM UTC

tptacek wrote: post #184

aerhardt wrote: post #34

My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question: "Is Paul Newman known for having had problems with alcohol?" All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response: "Paul Newman is not widely known for having had problems with alcohol. While he portrayed charact…

I just asked 4o: Yes, Paul Newman did experience significant struggles with alcohol. In his posthumously published memoir, The Extraordinary Life of an Ordinary Man, Newman candidly discusses his drinking habits and acknowledges his long-term battle with alcoholism. He describes himself as a "functioning alcoholic," a trait he noted was shared with his father. At one point, Newman was reported to consume a case of be…