Very much agree. All these currently hyped workflows removing the human from the loop attribute a modularity to those agents that just does not hold up. They will always be leaky abstractions given their stochastic nature.
That being said, they can be a great tool for getting past the "blank page" and just start or getting unstuck in general.
The Eternal Sloptember
351–360 of 389 posts
Re: The Eternal Sloptember
#352Part of my job is working on trying to make these models productive for the large corporation I work for. It's a lot of throwing tomatoes at a wall and to a degree I see the issue he is talking about output seemingly having a certain ceiling. At the same time in no part of his post is any code snippet or anything to latch on to of "the model performed poorly here when it should have done this" - this style of critici…
> How are we really supposed to grasp their actual capabilities when no one will actually cite specifically what mistakes they are making. The mistakes they make are pretty subtle. Coding with LLMs can be like that scene in Whiplash – , not quite my tempo, , downbeat on 18, , you’re rushing, , dragging, … Like yeah it produces working code almost always and the code usually does what you asked. And yet it makes you w…
Last week I helped a co-worker with some flaky tests, where the code and tests were generated from one of the models. While looking at how one of the tests work, I'd spotted a place in the code where a boolean condition was backwards in a way a human would never have written (and on top if it, there was a confidently-incorrect block comment above it so it was easy to assume it was correct) - so even if he'd fixed the test's flakiness, it would end up always failing instead of sometimes failing. He'd spent hours trying to figure out what was going on.
Re: The Eternal Sloptember
#353I think a lot of the problem with the current discourse is how black-and-white it is. Either you're a luddite or "ai pilled". In most cases, LLMs can get you 80-95% of the way, sometimes less, sometimes more. And heck, sometimes, it just gets you somewhere wrong. But it seems everyone is arguing about whether LLMs can be perfect software engineers in isolation running in a closet, and using that to say that LLMs do n…
"Either you're a luddite or "ai pilled"."
The Luddites were (violent) activists. They were more than just "non-believers"
Generally, those being labeled "Luddites" in today's "discourse" are people who dare to question the "AI" hype. GGenerally, these people are not activists
Re: The Eternal Sloptember
#354I think a lot of the problem with the current discourse is how black-and-white it is. Either you're a luddite or "ai pilled". In most cases, LLMs can get you 80-95% of the way, sometimes less, sometimes more. And heck, sometimes, it just gets you somewhere wrong. But it seems everyone is arguing about whether LLMs can be perfect software engineers in isolation running in a closet, and using that to say that LLMs do n…
"Either you're a luddite or "ai pilled"." The Luddites were (violent) activists. They were more than just "non-believers" Generally, those being labeled "Luddites" in today's "discourse" are people who dare to question the "AI" hype. GGenerally, these people are not activists
Semantic weakening is common in all languages, just as literally doesn't literally mean literally.
Re: The Eternal Sloptember
#355Earlier quoted context omitted.
>But it didn't give us the beautiful portraits and inspired lanscapes we expected, only millions of pictures of food. Here's a sample of my work using digital cameras, not a food picture in sight. https://flickr.com/photos/---mike---/albums/7217772029640662... The thing about having the ability to take effectively free photographs is that it really lets you experiment and learn the edges of what's possible. I was ins…
The comparison between AI and digital photography is a great one IMHO. I used to be deep into photography in the 00s, until I came to the conclusion that I was spending lots of money, spending hours trying to take the perfect picture and then hours reworking them on the computer... for what? Sure, I had fun while doing all of this, but to me art is about sharing. And given the sheer amount of pictures that were publi…
>it seems that your art is not seen by many
Oh, there's a story there about assumptions and bad UI. I had fairly large numbers, and tried to whittle down the thousands of photos I had posted to the ones that were favorited, and in the process erased everyone elses favorite tags, leading to rage quitting Flickr for a while. It's all now a mere hint of what it was.
Re: The Eternal Sloptember
#356Earlier quoted context omitted.
Goods are usually (although not always) inferior when made by a machine. A hand-crafted solid wood table is still superior to something from Ikea. Of course hand made tables are expensive. They service a sliver of the market. Ikea serves the rest of us who'd prefer not to eat off the floor. Fundamentally, Luddites didn't like being replaced by a machine. They were skilled workers, who used to have very desirable skil…
> Goods are usually (although not always) inferior when made by a machine. This is only true in the beginning, when machines are still primitive (e.g. first automatic looms). Nowadays machines mostly yield much better quality than any human can produce (e.g. automated welding, anything CNC controlled). Many things are only possible to build with machines (e.g. semiconductors). > A hand-crafted solid wood table is sti…
I know you said “mostly,” but wood is a huge outlier. It’s too heterogeneous to be accurately worked by machines at the level of accuracy that truly fine craftsmanship demands. To be clear, I’m talking about the level where a single chair can cost $20K, and represents hundreds of hours of labor. My FIL is at that level, and the stuff he makes is insane. The market for it is of course tiny, and I’d wager most of his buyers don’t even appreciate the attention to detail he puts into it, but yeah - there are no machines that I’m aware of that can feel perturbations at the sub-mm level and adjust the tool head on the fly, but he can.
This may simply be due to a lack of demand, but regardless, I assure you that machine-produced furniture can’t touch human-produced at the apex of fine craftsmanship.
Re: The Eternal Sloptember
#357AI labs should put some incentives in their RL to make their models write shorter code so it's easier to check
The issue then becomes it writes insane one liners that are impossible to read. Would be difficult to set up an RL environment where the goal is correctness and readability.
you could have an AI judge look for correct and readable code
Re: The Eternal Sloptember
#358Earlier quoted context omitted.
You could both be right.
No, I don't think people will be spending even more money on AI if it is not becoming productive. 2 years is a long time to get used to it.
You can be completely right in a bad way, and it's what I suspect is going to happen.
Three years ago I mentioned to co-workers I was most concerned about juniors not being able to build skills to become mid-level or senior. In the past year others have started talking about the same. But also I had thought people who were already mid-level or senior could resist and control themselves enough to use it well - but in the past six months two different co-workers have independently said they've noticed their own skills atrophying. And with those skills atrophied, they'll have less and less input/direction for the AI tools.
My suspicion is that for everyone who has gone all-in, a few years from now, they will still see a productivity increase from their own baseline - but their baseline will have dropped from where it is now as they "get used to it".
Re: The Eternal Sloptember
#359I'm in the "haven't written any code in a while" boat ATM. I'd love to see examples of issues that are so big that they warrant reverting to manual coding. My main issue has been the inconsistent quality across between model releases and the tendency to insert older APIs or documentation, especially with command line tools. I can understand if the model struggles with a million line monolithic codebase with a decade…
It isn't "so big." Things can also be "so small" that it's not worth it.
Coding isn't very hard, so it's often easier to just code than read and write English. I write Haskell exclusively though so this might bias me.
Re: The Eternal Sloptember
#360Earlier quoted context omitted.
It's the LLM version of "You're holding it wrong."
-> You Are Holding It Wrong. LLMs + Harnesses are incredibly effective, as evidenced by the literal millions of people who are paying quite a lot to use them, who speak glowingly of them and would 'never go back'. Whatever 'shape they take' - they are obviously useful - ergo - 'you're doing something wrong' if you can't make use of them for most tasks. Mileage varies, there are downsides, but it's the same with anyth…
> as evidenced by the literal millions of people who are paying quite a lot to use them, who speak glowingly of them and would 'never go back'.
Anecdotes from sunk cost users aren't evidence.