Reminder to not hire people who worked at MBB.
ChatGPT-4 significantly increased performance of business consultants
71–80 of 282 posts
Re: ChatGPT-4 significantly increased performance of business consultants
#72LLMs are stunningly good at language tasks: almost all of what us old-timers called NLP is just crushed these days. Summarization, Q&A, sentiment, the list goes on and on. Truly remarkable stuff. And where there isn’t a bright line around “fact”, and where it doesn’t need to come together like a Pynchon novel, the generative stuff is smoking hot: short-form fiction, opinion pieces, product copy? Massive productivity…
I wonder how long it takes AI to get good at law. Right now the verbal tasks it excels at are similar to the artistic ones: namely, solving problems with enormous solution spaces that are robust to small perturbations. That is, change a good picture of an angry tree man slightly and it's still probably a good picture of an angry tree man.
Re: ChatGPT-4 significantly increased performance of business consultants
#73Earlier quoted context omitted.
I’m starting to think there’s an LLM equivalent to the old saying about how everything the media writes is accurate except on the topics you’re an expert in. All LLM output looks to be good quality except when it’s output you’re an expert in. People who have no background in writing or editing think LLMs will revolutionize those fields. Actual writers and editors take one look at LLM output and can see it’s basically…
Programmers don't think that, though, or least not all the time. You could say similar things about Stack Overflow, and yet we use it.
For any managers reading: Chat GPT and Stack Overflow are not the same kind of thing.
Re: ChatGPT-4 significantly increased performance of business consultants
#74Well, this sounds like perfect tasks for GPT: "Participants responded to a total of 18 tasks (or as many as they could within the given time frame). These tasks spanned various domains. Specifically, they can be categorized into four types: creativity (e.g., “Propose at least 10 ideas for a new shoe targeting an underserved market or sport.”), analytical thinking (e.g., “Segment the footwear industry market based on…
> A confident GPT hallucination is almost indistinguishable from typical management consulting material...
If you're measuring based on output, sure, but... the value of any knowledge worker is primarily driven by the input, that is, a client doesn't want "10 ideas" they want "10 [valuable] ideas [informed by the understanding of the business and the market they're operating in]". If a management consultant said "boat shoes" in response to this question they would not have a client much longer.
You could apply this same nonsense task to software engineering, i.e: ask ChatGPT to "write 10 lines of code" and it'll be indistinguishable from the code we churn out day after day.
Re: ChatGPT-4 significantly increased performance of business consultants
#75HN is so bad at predictions. Just a few months ago HN was awash with comments that confidently claimed LLMs were no more than stochastic parrots and unlikely to amount to anything. > I can't help but think the next AI winter is around the corner. [0] Yeah, right. [0] https://news.ycombinator.com/item?id=23886325
Who claimed management consultants are not stochastic parrots?
Re: ChatGPT-4 significantly increased performance of business consultants
#76Earlier quoted context omitted.
Yep. ChatGPT is like having a junior engineer confidently asking to merge broken garbage into your codebase all the time. Adds negative value for anyone that knows what they’re doing.
hahahah. A friend of mine has a problem with a contractor at his workplace that tries to PR in shell scripts written with Copilot. My friend spends an hour to explain why a script generated in 5 minutes is horrifically awful and will likely take down the company. He's legitimately angry about it.
It seems like the only ways to subordinate programming tasks are to write tests for your subordinate's code, or to review it tediously yourself, or to just trust the hell out of them.
Re: ChatGPT-4 significantly increased performance of business consultants
#77More details in this blog post by a Wharton professor: https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the... My questions to naysayers: * Do you or anyone you know use GPT-4 (not the free GPT-3.5) to do productive tasks like coding and found it to help in many cases? * If you insist it’s useless, why do millions of people pay $20 a month to access GPT-4 and plugins?
Yes, GPT-4 is great for doing “boring work” and allows me to focus on the “fun work”. You still need to know what you’re doing though, you can’t blindly copy and paste.
And for the second one, although I am paying for it too, this idea is more or less flawed nowadays. Utilization is a very hand wavy thing when it comes to this stuff. Like a purse, millions would pay money for it, some even pay thousands. But I have no use for it and wouldn’t even pay a $1 for one.
Re: ChatGPT-4 significantly increased performance of business consultants
#78Well, they buried the lede with this one. Using LLMs were better for some tasks and actually made it worse for others.
The first task was a generalist task ("inside the frontier" as they refer to it), which I'm not surprised has improved performance, as it purposely made to fall into an LLM's areas of strength: research into well-defined areas where you might not have strong domain knowledge. This also is the mainstay of early consultants' work, in which they are generalists in their early careers – usually as business analysts or similar – until they become more valuable and specialise later on.
LLMs are strong in this area of general research because they have generalised a lot of information. But this generalisation is also its weakness. A good way to think about it is it's like a journalist of research. If you've ever read a newspaper, you often think you're getting a lot of insight. However, as soon as you read an article on an area of your specialisation, you realise they've made many flaws with the analysis; they don't understand your subject anywhere near the level you would.
The second task (outside the frontier) required analysis of a spreadsheet, interviews and a more deeply analytical take with evidence to back it up. These are all tasks that LLMs aren't strong at currently. Unsurprisingly, the non-LLM group scored 84.5%, and between 60% and 70.6% for LLM users.
The takeaway should be that LLMs are great for generalised research but less good for specialist analytical tasks.
Re: ChatGPT-4 significantly increased performance of business consultants
#79LLMs are stunningly good at language tasks: almost all of what us old-timers called NLP is just crushed these days. Summarization, Q&A, sentiment, the list goes on and on. Truly remarkable stuff. And where there isn’t a bright line around “fact”, and where it doesn’t need to come together like a Pynchon novel, the generative stuff is smoking hot: short-form fiction, opinion pieces, product copy? Massive productivity…
I wonder how long it takes AI to get good at law. Right now the verbal tasks it excels at are similar to the artistic ones: namely, solving problems with enormous solution spaces that are robust to small perturbations. That is, change a good picture of an angry tree man slightly and it's still probably a good picture of an angry tree man.
Depends what you would classify as good, but IBM Watson is already used in law firms [today](https://www.ibm.com/case-studies/legalmation)
LLMs iare most often best at helping humans do their tasks more effectively, not replacing them completely
Re: ChatGPT-4 significantly increased performance of business consultants
#80More details in this blog post by a Wharton professor: https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the... My questions to naysayers: * Do you or anyone you know use GPT-4 (not the free GPT-3.5) to do productive tasks like coding and found it to help in many cases? * If you insist it’s useless, why do millions of people pay $20 a month to access GPT-4 and plugins?
I have free access to copilot because I do some open source work. I haven't been impressed by what it can do and I wouldn't pay even $3/month to use it.
The second question doesn't make sense to me. There are tons of things I think are useless (or worse) that people pay for anyway. Meal kit boxes come to mind, and at least you can eat those at the end of the day.