761–770 of 1001 posts
It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
Earlier quoted context omitted.
When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…
while I agree that LLMs do not have thoughts or plan. They are merely text generators. But when you give the text generator ability to make decisions and take actions, by integrating them with real world, there are consequences. Imagine, if this LLM was inside a robot, and the robot had ability to shoot. Who would you blame?
It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
>> the vast majority of an engineer's time isn't spent writing -- it's spent reading and thinking.
Unfortunately, this is now an extremely minority understanding of how we need to do our job - both among hirees and the people who hire them. You're lucky if you can find an employer who understands the value of it. But this is what makes a "10x coder". The unpaid time spent lying awake in bed, sleepless until you can untangle the real logic problems you'll have to turn into code the next day.
It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…
When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…
Sometimes those actions are "e-mail a customer back", other times they are "submit a new pull request on some github project" and "file a new Jira ticket." Other times the action might be "blackmail an engineer."
Not saying it's time to freak out over it (or that it's not time to do so). It's just weird to see people go "don't worry, token generators are not experiencing subjectivity or qualia or real thought when they make insane tokens", but then the tokens that come out of those token generators are hooked up to executable programs that do things in non-sandboxed environments.
It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year
It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year
40% to 80% is a 2x improvement
It’s not that the second leap isn’t impressive, it just doesn’t change your perspective on reality in the same way.
After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solving/debugging if you are expecting accurate results and correct code. However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of tha…
I've noticed an interesting trend: Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details. Most people who are not happy with LLM coding ... provide almost no details. As someone who's impressed by LLM coding, when I read a post like yours, I tend to ha…
1) Writing a high-performance memory allocator for a game engine in Rust: https://github.com/golddranks/bang/tree/main/libs/arena/src (Still work in progress, so it's in a bit messy state.) Didn't seem to understand the design I had in mind, and/or the requirements and goes on tangents and starts changing the design. In the end, coded the main code myself and used LLM for writing tests with some success. Had to remove tons of inane comments that didn't provide any explanatory value.
2) Trying to fix a Django ORM expression that generates unoptimal and incorrect SQL. Constantly changes opinion whether something is even possible or supported by Django, apologizes when pointing out mistakes / bugs / hallucinations, but then proceeds to not internalize the implications of the said mistakes.
I used the Zed editor with its recently published agentic features. I tried to prompt it with a chat style discussion, but it often did bigger edits I would have liked, and failed to share a high-level plan in advance, something I often requested.
My biggest frustrations were not coding problems per se, but just general inability to follow instructions and see implications, and lacking the awareness to step back and ask for confirmations or better directions if there are "hold on, something's not right" kind of moments. Also, generally following through with "thanks for pointing that out, you are absolutely right!" even if you are NOT right. That yes-man style seriously erodes trust in the output.