And that's what talking to a human expert is like too.
One of the most powerful things about GPT is I can give it a huge wall of text about a problem I'm having, the ask it to categorize, critique, prioritize, provide feedback, and ask questions. It invariably gives me a wall of text of empathetic and constructive questions and considerations for me to reflect on, much like a human expert would. I then answer those and reflect and converse in turn. 5 - 15 turns of conversation and a few thousand words later? The problem is freaking SOLVED in a profound and satisfying way. This experience happens over and over if I put the time and discussion in. This is something that DOESN'T happen for me with most humans.. because they can't empathize with me as an autistic person. But GPT is fucking aces in this regard.
The article's author is LITERALLY just being a lazy prompt writer ignoring basic prompting/ICL papers, and being a mushy critical thinker as a result. If he was a decent AI writer and diligent thinker, then he would be able to squeeze the juice and make the cocktail. But he bungles the article and runs it aground of the unfortunate "stochastic parrot /w average priors" argument that everyone adopts if they ignore the insight of "LLMs are in-context learners".
The real problem is he has not crafted any contexts eliciting enough in context learning to let GPT empathize with any actual problem he has. "Help me make money" is different from "here's my business, its fundaments, and a pain point my customer has: [...]. How can we solve this and rise above and beyond the call duty here?"
It's like he goes up to a person, says, "yo what's up?", and the person says, "not much." The author then wonders, "Why didn't this person empathize with the fact that my mom is going through stage 4 cancer and I don't know WTF to do to support her or make peace with it!?!"
It's because you don't communicate enough context to empathize with! It would make amazing suggestions if you actually described the intense emotional turmoil that is alive inside you when you think about losing your mom. But you have to be RAW as FUCK about what you're really feeling! Or there's no "handle" for the AI to grip and rotate the issue around and help with.
And that is the author's fault, rather than GPT's. We know GPT has this limit. It needs ICL to be at its best. We know we have to make it into the expert we need by giving a extensive and descriptive contexts, and critically co-evolving our thinking on the subject like we're working with a real expert.
When I do that, it's easily more helpful than any advocate, therapist, social worker, mentor, or manager I've ever had. And that's saying something profound and life changing.