Earlier quoted context omitted.
Hey Tombert, wrt did you read the article? I was quite specific about the ways I think LLMs are blurring the lines. I don't think its true for general engineering but I do think its true for applications being built with LLMs. Also its still very early
It's not early. It has reached a plateau. Are there betting odds for "AI" (LLM) benchmarks somewhere? I will bet money
AI is blurring the line between PMs and engineers?
41–50 of 63 posts
Re: AI is blurring the line between PMs and engineers?
#42> “No, no,” she said. “Engineers aren’t allowed to edit the prompts. It’s only the PMs and domain experts who do prompt engineering. They do it in a custom UI, and then the prompts are committed to the codebase.”
This feels like a desperate power grab.
Why can’t engineers be involved with the prompts? Why aren’t they allowed to do things like automated A/B testing or to implement ideas from papers they’ve read?
Banning engineers from prompts altogether feels extremely arbitrary. If this was a healthy relationship they’d at least say the engineers and PMs work together on prompts.
Banning them is just politics.
Re: AI is blurring the line between PMs and engineers?
#43Depends. If it is for writing documents, you really don't need a lot of 'technical writers'. For managing engineers? Honestly a competent staff engineer can replace 2 PMs. For writing code with PMs replacing Engineers with AI assistants? Very unlikely, even if they try as soon as you get to the question on maintaining an existing codebase, this is where LLMs begin to hallucinate more and struggle. The typical PM won'…
Most engineers are bad at writing documentation. They are so blind at writing badly that they won't even realize their writing is bad. I've reviewed a large number of design docs, and the average quality is so bad that the doc reviewer will just reply with many comments and requests for clarification. AI can fix things like grammar and phrasing, but it won't be able to fix the context that's missing.
Managing engineers is not the job of PMs anyway. PMs are supposed to manage projects, not people.
Re: AI is blurring the line between PMs and engineers?
#44A joke: Good PMs will do invisible work. Bad PMs are just invisible.
As anyone who has worked with a bad PM can tell you, they are anything but invisible.
Re: AI is blurring the line between PMs and engineers?
#45Earlier quoted context omitted.
Hey Tombert, wrt did you read the article? I was quite specific about the ways I think LLMs are blurring the lines. I don't think its true for general engineering but I do think its true for applications being built with LLMs. Also its still very early
I'm just saying that I don't think I agree with some of this; even if PMs are writing the prompts (and calling that "prompt engineering"), it's not equivalent because they don't know how to audit the code given to them. A PM might generate that SQL thing I mentioned and just blindly cut and paste it. For any application with more than one user, that is a bug, it's incorrect, and it's not like this is some deep cut: u…
My read–which may be wrong–is that much of the article is discussing applications where the end user is interacting with an interface that queries an LLM using baked-in prompts (in one case, a marketing content generation tool). These prompts are being written by the PM. The PM is not writing prompts LLMs to generate code, the PM is writing prompts which are hidden behind a web form or button or something in an interface, hence the prompts being part of the codebase. The author argues that when a PM is editing these prompts they are delivering an artifact that looks more like an engineer's artifact than a PM's artifact, traditionally.
Re: AI is blurring the line between PMs and engineers?
#46No mention of Data Scientists (which is neither PM
or Eng), seems odd.
Re: AI is blurring the line between PMs and engineers?
#47Earlier quoted context omitted.
Hey that's a topic I can get behind! Most of the big CEOs appear to be CEO of multiple companies (Musk and Bezos come to mind). If you can be the CEO of five different companies, it really doesn't seem like it can be that hard of a job. ChatGPT might be able to do that just fine.
> Most of the big CEOs appear to be CEO of multiple companies (Musk and Bezos come to mind). Most big CEOs are definitely not in charge of multiple companies. You hear about Musk and Bezos because they’re all over the news, but they’re all over the news because they’re not normal CEOs. It’s well known that Musk’s relationship to his companies is primarily one of ownership and delegation. Bezos hasn’t been in charge o…
> It’s funny how often I hear this about AI replacing jobs, with one exception: Everyone who repeats it is highly confident that AI can replace other jobs but once you get to the work they do they’ll stop and explain why it’s actually much harder than you think.
Call it the Turing-Lovelace Amnesia Effect.
Re: AI is blurring the line between PMs and engineers?
#48Forget PMs and engineers, the discussion I want to see happening is just how much more easily an LLM could replace the average CEO...
Real-talk: using an LLM for CEO-work makes sense for when hard, rational, decisions need to be made (e.g. new divisions, layoffs, expansion, etc) - but CEOs also need to be a leader and inspire confidence in their subordinates - a non-human agent with non-human motivations simply can’t do that.
I'd expect that AI making top-level decisions would be disastrous for employees, especially in tech. An AI is going to relentlessly push for efficiency, and human talent simply can't be reduced to a single input to measure against cost (salary, benefits, devtools, etc). AI will constantly suggest lower costs and thus fewer people, it has no perception of product quality or satisfaction, and it's always going to question the value of the employees' contributions to building software (or other stuff), especially if those employees are themselves relying on AI assistance.
Re: AI is blurring the line between PMs and engineers?
#49Forget PMs and engineers, the discussion I want to see happening is just how much more easily an LLM could replace the average CEO...
A theme of pre-LLM computer-human hybrid systems, was one expensive/scarce expert might be replaced by a less-expensive computer-facilitated redundant group of lesser experts.
So perhaps imagine something like Kaggle competitions, but for Harvard Business School case studies. Open to LLMs, humans, and collaboratives.
A first step might be to create a leadership/managerial LLM test set. I wonder if HBS case studies are in training sets. And whether we can generate good case studies yet. Perhaps use military leadership training material?
Re: AI is blurring the line between PMs and engineers?
#50Forget PMs and engineers, the discussion I want to see happening is just how much more easily an LLM could replace the average CEO...
Real-talk: using an LLM for CEO-work makes sense for when hard, rational, decisions need to be made (e.g. new divisions, layoffs, expansion, etc) - but CEOs also need to be a leader and inspire confidence in their subordinates - a non-human agent with non-human motivations simply can’t do that.
> using an LLM for CEO-work makes sense for when hard, rational, decisions need to be made (e.g. new divisions, layoffs, expansion, etc)
Hard disagree. Using an LLM for these decisions transforms it into a game of manipulating inputs.
If you thought it was bad when people were gaming metrics for performance reviews, imagine the nightmare of a company where everyone is trying to manipulate their work to appeal to the current preferences of the prompts HR is feeding to the LLM. Now imagine the HR people manipulating the prompts to get the answers they want, or leaking the prompts to their friends.
Without humans in the loop to recognize when incentives are being gamed it becomes a free for all.