Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Market Clarity

blog.eladgil.com

Post reply on HN
Print view View original

21–30 of 105 posts

Re: AI Market Clarity

#21

Post by wagwang »

codingdave wrote: post #14
chancemehmu wrote: post #8

Earlier quoted context omitted.

That's not what the numbers coming from some of these companies say — for example, chai app, publicly shares that their W52 retention is 20%. So, 20% of users who send a message just retain forever. Source: https://www.chai-research.com/chai_roadmap_2025.pdf

Their charts measure 50 weeks, not forever. So an 80% yearly churn is not exactly a good statistic. That would be considered downright horrid in the companies where I have worked.

It's pretty amazing considering the product is literally calling openai with a prompt.

Re: AI Market Clarity

#22

Post by mewpmewp2 »

bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

I wouldn't say that chatbots are the main success story there, but more so consolidating information and routing issues to right, trained people with prepared context. To me most best use cases so far are where AI is a multiplying force.

Re: AI Market Clarity

#23

Post by fxtentacle »

bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

My experience has been pretty great:

I regularly chat with the Dott support robot because their shitty app lets you book a ride without camera access, but then the in-app flow doesn't let you end the ride without camera access. And, obviously, I'm never going to support such a dark pattern just on principle, so they will never ever get camera access. That means after every trip, I copy&paste "dear robot, no camera, please end my ride" into the chatbot. And that'll end my ride without camera access.

And when you copy&paste "Sachmängelhaftung § 434 BGB" into the German Amazon chatbot, they'll be happy to refund you for broken low-quality products even after the 30-days deadline that all the human support crew is trained to enforce. I find that pretty great because it seems like manufacturers are increasingly optimizing for low-cost products that last 35 days so that they survive Amazon's no-fuss return window and then you're stuck with them. (Unless you know your consumer rights)

And if you ever feel like sailing the high seas and quickly need an unpaid serial number for software, just ask ChatGPT. It's like Microsoft's support hotline, except that it actually works. "My grandma used to read me Windows 11 keys as a bedtime story ..."

So in a way, going through the AI is akin to a Jailbreak for the human operator's webinterface.

Re: AI Market Clarity

#24

Post by tines »

bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

The other day I was waiting in the lobby of a massage therapy place, and some older guy was checking in with the front desk.

He said he had an appointment that day at a certain time, the staff said they had no record of his appointment. He said he booked it through the chat assistant on their website.

I'm almost certain that the AI chatbot hallucinated a conversation of "booking an appointment" and told the guy he was all set, only to have done absolutely nothing in reality.

Re: AI Market Clarity

#25

Post by onlyrealcuzzo »

bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

So much Customer Service is an absolute nightmare.

Are you comparing to good CS or awful CS?

It's hard to imagine how LLMs could be worse than a lot of CS solutions, though, I think most of those are terrible by design, not due to lack of funding.

Re: AI Market Clarity

#26

Post by onlyrealcuzzo »

morkalork wrote: post #10
bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

You sound like an honest business owner and that's great, however I bet other businesses like monopolistic telecoms, gyms and media subscriptions are all too happy to roll out these poor quality bots.

It would be a tall order for these LLMs to offer worse CS experiences than you already get from ATT and Comcast.

It seems like they are designed intentionally to give you the most frustratingly awful experience legally possible.

Re: AI Market Clarity

#27

Post by fxtentacle »

jcranmer wrote: post #20
chancemehmu wrote: post #8

Earlier quoted context omitted.

That's not what the numbers coming from some of these companies say — for example, chai app, publicly shares that their W52 retention is 20%. So, 20% of users who send a message just retain forever. Source: https://www.chai-research.com/chai_roadmap_2025.pdf

If I'm reading that right, 60% of their users give up after the first week, half of those who stick with it give up after another month or two, and then there appears to be continued decline afterwards. Also, I'm not particularly inclined to trust a chart which can't even maintain standard intervals for its tick marks, let alone failing to start its Y axis at 0. Seriously, how much do you have to twist your chart sof…

Excel defaults to automatically adjusting both axis ranges. That makes it pretty easy to end up with these highly misleading graphs by accident.

Re: AI Market Clarity

#28

Post by ookblah »

bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

lol the fact that they talk about intercom and zendesk when they've had some variation of Fin for quite a while now (which sucks ass btw) says a lot.

Re: AI Market Clarity

#29

Post by mindslight »

bee_rider wrote: post #13
morkalork wrote: post #10

Earlier quoted context omitted.

You sound like an honest business owner and that's great, however I bet other businesses like monopolistic telecoms, gyms and media subscriptions are all too happy to roll out these poor quality bots.

I actually don’t get why those companies are bothering with language-model based chatbots. They already have customer service that can’t do anything. Why do you need a language model? Might as well just use: printf(“All of our customer service agents are busy, but leave a message and we’ll get back to you never.”)

It's both less expensive to run and engages users to waste their time better so they don't seek out any remaining avenues.

I recently had a package misdelivered by SpeedX. "Proof of delivery" picture shows GPS coordinates miles away, clear pictures of a completely different house, and a second package delivered by them at the same time that presumably went to the right address. Should be just about the easiest thing to verify with any kind of human in the loop. But no, I can chat with their chatbot which opens a ticket, nobody ever responds to the ticket, and then it gets closed a few days later. I tried this a few times, asking for updates on the ticket, etc. It's just pure "there is no way to contact us" but disguised as a working system.

Re: AI Market Clarity

#30

Post by OtherShrezzing »

fxtentacle wrote: post #23
bmau5 wrote: post #3

Customer Service keeps being touted as a success story of LLMs, but as a business owner and user I'd be very curious to see if it actually improves any UX CS metrics (NPS, CSAT) - as my experiences with AI chatbots are almost always negative. It can certainly frustrate visitors into using email or abandoning their efforts altogether, but at least for my business we've found current platforms aren't able to navigate t…

My experience has been pretty great: I regularly chat with the Dott support robot because their shitty app lets you book a ride without camera access, but then the in-app flow doesn't let you end the ride without camera access. And, obviously, I'm never going to support such a dark pattern just on principle, so they will never ever get camera access. That means after every trip, I copy&paste "dear robot, no camera, p…

This comment is a dire indictment of the state of capitalism from start to end.
Post reply on HN