Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

95% of Companies See 'Zero Return' on $30B Generative AI Spend

thedailyadda.com

Post reply on HN
Print view View original

381–390 of 445 posts

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#381

Post by aaomidi »

la_fayette wrote: post #276
doorhammer wrote: post #205

Earlier quoted context omitted.

Sentiment analysis, nuanced categorization by issue, detecting new issues, tracking trends, etc, are the bread and butter of any data team at a f500 call center. I'm not going to say every project born out of that data makes good business sense (big enough companies have fluff everywhere), but ime anyway, projects grounded to that kind of data are typically some of the most straight-forward to concretely tie to a dol…

Yes that sound like important and useful use cases. However, these are solved by boring old school ML models since years...

Sentiment analysis was not solved and companies were paying analyst firms shit tons of money to do that for them manually.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#382

Post by krainboltgreene »

ch4s3 wrote: post #209
krainboltgreene wrote: post #168

Earlier quoted context omitted.

Pro-tip: You won't ever do that.

I would imagine OP is probably mining service call summaries to find common service issues, or at least that's what I would do.

That's what everyone says they'll do and then it never gets touched again.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#383

Post by pepinator »

eldenring wrote: post #29

This is how America ends up being ahead of the rest of world with every new technology breakthrough. They spend a lot of money, lose a lot of money, take risks, and then end up being too far for others to catch up. Trying to claim victory against AI/US Companies this early is a dangerous move.

That's an awfully simplistic way of seeing things. It always surprises me how disconnected from reality Americans are.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#384

Post by doorhammer »

Imustaskforhelp wrote: post #358
doorhammer wrote: post #294

Earlier quoted context omitted.

Again, not the OP, so I can't speak to exactly their use-case, but the vast majority of call center calls fall into really clear buckets. To give you an idea: Phonetic transcription was the "state of the art" when I was a QA analyst. It broke call transcripts apart into a stream of phonemes and when you did a search, it would similarly convert your search into a string of phonemes, then look for a match. As you can i…

I genuinely don't think that the GP is actually making someone actually listen to the transcription and summary and check if the summary is wrong. I almost have this gut feeling that its the case (I may be wrong though) Like imagine this, if the agent could just spend 3 minutes writing a summary, why would you use AI to create a summary and then have some other person listen to the whole audio recording and check if…

They might not be, and their use-case might not be one I agree with. I can just imagine a plausible reality where they made a reasonable decision given the incentives and constraints, and I default to that.

I'm basically inferring how this would go down in the context I worked under, not the GP, because I don't know the details of their real context.

I think I'm seeing where I'm not being as clear as I could, though.

I'm talking about the lifecycle of a methodology for categorizing calls, regardless of whether or not it's a human categorizing them or a machine.

If your call center agent is writing summaries and categorizing their own calls, you still typically have a QA department of humans that listen to a random sample of full calls for any given agent on a schedule to verify that your human classifiers are accurately tagging calls. The QA agents will typically listen to them at like 4x speed or more, but mostly they're just sampling and validating the sample.

The same goes for _any_ automated process you want to apply at scale. You run it in parallel to your existing methodology and you randomly sample classified calls, verifying that the results were correct and you _also_ compare the overall results of the new method to the existing one, because you know how accurate the existing method is.

But you don't do that for _every_ call.

You find a new methodology you think is worth trying and you trial it to validate the results. You compare the cost and accuracy of that method against the cost and accuracy of the old one. And you absolutely would often have a real human listen to full calls, just not _all_ of them.

In that respect, LLMs aren't particularly special. They're just a function that takes a call and returns some categories and metadata. You compare that to the output of your existing function.

But it's all part of the: New tech consideration? -> Set up conditions to validate quantitatively -> run trials -> measure -> compare -> decide

Then on a schedule you go back and do another analysis to make sure your methodology is still providing the accuracy you need it to, even if you haven't change anything

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#385

Post by krainboltgreene »

ninininino wrote: post #200
krainboltgreene wrote: post #168

Earlier quoted context omitted.

Pro-tip: You won't ever do that.

Advanced organizations (think not startups, but companies that have had years of decades of profit in the public market) might have solved all the low-hanging fruit problems and have staff doing things like automated quality audits (search summaries for swearing, abusive language, etc).

I've worked at both. It is extremely rare that anyone ever does it.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#386

Post by missedthecue »

tux3 wrote: post #370
missedthecue wrote: post #365

Earlier quoted context omitted.

What i'm saying is that if the volume of support is high enough, and never even changed, it's completely possible to improve throughput without reducing demand for labor. The result is simply that you improve response times.

But I think this comes back to the same question of understaffing/overwork. We have to ask what strategic thinking led to accept long response times in the past. And the answer is unequivocal. Unless we're claiming there is an intractable qualified labor shortage in call centers, this is always the result of a much simpler explanation: it's much cheaper to understaff call centers A company that wants to save money by…

OK, but in that case, we reach status quo but with fewer employees. Doesn't that meet your definition of efficiency gains?

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#387

Post by actsasbuffoon »

Imanari wrote: post #229
actsasbuffoon wrote: post #144

Earlier quoted context omitted.

That’s the thing. There’s value in AI, it’s just not worth half a trillion dollars to train a new model that’s 0.4% better on benchmarks. Meta is never going to get a worthwhile return on spending $100M on individual engineers. But that doesn’t mean AI is without its uses. We’re just in that painful phase where the hype needs to die down and we treat LLMs as what they really are; an interesting new tool in the toolki…

Could you broadly describe the AI projects you have built?

I literally just had a conversation with my CEO this morning where he told me not to disclose the projects I’ve been working on, so I can only speak about it obliquely.

We identified some problems our customers have, and I’ve come up with interesting ways to use LLMs as part of an automated system to solve some of those problems. It’s not the kind of thing where we just dump some data into the ChatGPT API and get an answer. We’re doing fairly deep integrations that do some interesting/powerful things. It’s been a big deal for our prospective clients and investors.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#388

Post by doorhammer »

sillyfluke wrote: post #354
doorhammer wrote: post #294

Earlier quoted context omitted.

Again, not the OP, so I can't speak to exactly their use-case, but the vast majority of call center calls fall into really clear buckets. To give you an idea: Phonetic transcription was the "state of the art" when I was a QA analyst. It broke call transcripts apart into a stream of phonemes and when you did a search, it would similarly convert your search into a string of phonemes, then look for a match. As you can i…

Thanks for the detailed domain-specific explanation, if we assume that some whale clients of the company will end up in the call center is it not more probable that more competent human agents will be responsible for the call, whereas it's pretty much the same AI agent adressing the whale client as the regular customers in the alternative scenario?

Yeah, if I were running a QA department I wouldn't let llms anywhere near actual customers as far as trying to resolve a customer issue directly.

And, this is just a guess, but it's not uncommon that whale customers like that have their own dedicated account person and I'd personally stick with that model.

The use-case I'm like "huh, yeah, I could see that working well" is mostly around doing sentiment analysis and call tagging--maybe actual summaries that humans might read if I had a really well-design context for the llm to work within. Basically anything where you can have an acceptable false positive/negative rate.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#389

Post by Spivak »

thfuran wrote: post #271
Spivak wrote: post #174

Earlier quoted context omitted.

Why wouldn't it, once you actually have that project you have the raw audio to generate the transcripts. Only spend the money at the last second when you know you need it. Edit: Tell me more how preemptively spending five figures to transcribe and summarize calls in case you might want to do some "data engineering" on it later is a sound business decision. What if the model is cheaper down the road? YAGNI.

A company that could save millions by not having staff write up their own call notes almost surely is already doing that.

And yet the topic of conversation is a company that did just that. The AI is just the smoke and mirrors that pushed the business to do it. Staff aren't writing their own call notes anymore. The LLM summary, almost by definition, isn't adding any additional signal to the call audio. If your data engineering pipeline works by processing LLM generated notes then it must work equally well processing the call transcript—they're the exact same data. AI finally got the business to admit that nothing of value was added by call notes and have dropped that work completely. The final step is just dropping the useless use of LLM.

Just the audio transcript is way cheaper and can use existing technology.

Re: 95% of Companies See 'Zero Return' on $30B Generative AI Spend

#390

Post by cube00 »

vosper wrote: post #127
Shank wrote: post #99

Earlier quoted context omitted.

Who reads the summaries? Are they even useful to begin with? Or did this just save everyone 3-5 minutes of meaningless work?

AI reads them and identifies trends and patterns, or answers questions from PMs or others?

AI writes inaccurate summaries and then consumes its own slop so it can hallucinate the answer to the PM's questions after misreading said slop.

Much like dubbing a video tape multiple times, it's going to get worse as you add more layers text predictors.

Post reply on HN