The threat is if you replace your cognitive capabilities with AI, but you don't control entire the system your AI runs on (hardware, firmware, drivers, OS, weights, frontend), then that's equivalent to someone else owning a part of your brain.
The threat is comfortable drift toward not understanding what you're doing
361–370 of 668 posts
Re: The threat is comfortable drift toward not understanding what you're doing
#362[deleted]
Re: The threat is comfortable drift toward not understanding what you're doing
#363The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…
> So if Bob can do things with agents, he can do things.
Yes, but how does he know if it worked? If you have instant feedback, you can use LLMs and correct when things blow up. In fact, you can often try all options and see which works, which makes it ”easy” in terms of knowledge work. If you have delayed feedback, costly iterations, or multiple variables changing underneath you at all times, understanding is the only way.
That’s why building features and fixing bugs is easy, and system level technical decision making is hard. One has instant feedback, the other can take years. You could make the ”soon” argument, but even with better models, they’re still subject to training data, which is minimal for year+ delayed feedback and multivariate problems.
Re: The threat is comfortable drift toward not understanding what you're doing
#364Earlier quoted context omitted.
Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.
People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…
[deleted]
Re: The threat is comfortable drift toward not understanding what you're doing
#365I know how we can fix this....
Its of course devious, exactly some of our styles :)
Give AI to VCs to use for all their domain stuff....
They than make wrong investment decisions based on AI wrong info and get killed in the market....
Market ends up killing AI outright....problem solved temporarily
Re: The threat is comfortable drift toward not understanding what you're doing
#366Earlier quoted context omitted.
I would love to see someone attempt to do multiplication who never learned addition, or exponentiation without having learned multiplication. There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.” I learned how to do long division in school, decades ago. I sat down and tried it last year, and found myself struggling, because I hadn’t needed to do it in such a long time.
Most people learn multiplication by memorizing a series of cards 2x2,2x3.. 9x9. Later this gets broken down to addition in higher grades.
That’s definitely not how I learned it, nor how my kids have learned it. I vividly remember writing out “2 x 3 = 2 + 2 + 2 = 3 + 3.” I later memorized the multiplication table up to 12, yes, but that was not a replacement of understanding what multiplication was
Re: The threat is comfortable drift toward not understanding what you're doing
#367The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…
> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…
> The problem arrises when Bob encounters a problem too complex or unique for agents to solve.
It’s actually worse than that: the AI will not stop and say ”too complex, try in a month with the next SOTA model”. Rather, it will give Bob a plausible looking solution that Bob cannot identify as right or wrong. If Bob is working on an instant feedback problem, it’s ok: he can flag it, try again, ask for help. But if the error can’t be detected immediately, it can come back with a vengeance in a year. Perhaps Bob has already gotten promoted by then, and Bobs replacement gets to deal with it. In either case, Bob cannot be trusted any more than the LLM itself.
Re: The threat is comfortable drift toward not understanding what you're doing
#368Earlier quoted context omitted.
There's definitely folks working on automatically marketing via LLMs, but I have my doubts that it wont just numb people further to marketing as we are close to saturation.
My thought/worry on a lot of the LLM agentic workflow personal assistant stuff is it is just ripe for fraud. The money is more on the adversarial side. People think they'll just have a personal bot out there buying airline tickets, hotel rooms, jeans, new phone, etc. Meanwhile as soon as you have agents like this out in the wild, the capital will flow to bad actors creating bots to game those bots. The world is PvP u…
Yes it feels a lot like the earlier internet days, when people only saw the upside/utopian view based on a high trust environment.
Re: The threat is comfortable drift toward not understanding what you're doing
#369> The problem isn't that we'll decide to stop thinking. The problem is that we'll barely notice when we do
Most of what we call thinking is merely to justify beliefs that emotionally make us happy and is not creative per-se. I am making a distinction between "thinking" as we know it and "creative thinking" which is rare, and can see things in an unbiased manner breaking out of known categories. Arguably, at the PhD level, there needs to be a new ideas instead of remixing the existing ones.
Re: The threat is comfortable drift toward not understanding what you're doing
#370Earlier quoted context omitted.
This whole argument can be made for why every programmer needs to deeply understand assembly language and computer hardware. At a certain point, higher level languages stop working. Performance, low level control of clocks and interrupts, etc. I’m old enough dropping into assembly to be clever with the 8259 interrupt controller really was required. Programmers today? The vast majority don’t really understand how any…
Teaching how computer hardware works is pretty smart. There is no need to do it in depth though. Writing assembly is probably completely irrelevant. You should still know how programming language concepts map to basic operations though. Simple things like strict field offsets, calling conventions, function calls, dynamic linking, etc.
> Writing assembly is probably completely irrelevant.
ffmpeg disagrees.
More broadly, though, it’s a logical step if you want to go from “here’s how PN junctions work” to “let’s run code on a microprocessor.” There was a game up here yesterday about building a GPU, in the same vein of nand2tetris, Turing Complete, etc. I find those quite fun, and if you wanted to do something like Ben Eater’s 8-bit computer, it would probably make sense to continue with assembly before going into C, and then a higher-level language.