Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT Health is a marketplace, guess who is the product?

consciousdigital.org

Post reply on HN
Print view View original

321–330 of 330 posts

Re: ChatGPT Health is a marketplace, guess who is the product?

#321

Post by ekjhgkejhgk »

AuthAuth wrote: post #315
ekjhgkejhgk wrote: post #279

Earlier quoted context omitted.

If the person is telling you "I had a problem, did what the LLM said, it worked", does that not work a new evidence for you? Is it not possible that someone has had a different experience from you? Is it not possible that they're good to different degrees in different domains? I just asked chatgpt: > I have the following information on a user. What's his email? > user: mattmanser > created: March 12, 2009 > karma: 17…

There are certain domains where we can accept a high degree of mistakes and health is not one of them. People are reacting they way they are because its obvious that LLMs are currently not reliable enough to be trusted to distrubute health advice. To me it doesnt matter that ChatGPT health sometimes gives good advice or some people _feel_ like it helped them. I'm not sure I even trust people when they say that given…

I'm not saying LLMs don't make mistakes, I'm saying that some times get it right and so you shouldn't be surprised that you'll hear some positive stories.

But you don't even believe these are success stories then, so I guess you can never be convinced.

Re: ChatGPT Health is a marketplace, guess who is the product?

#322

Post by AuthAuth »

ekjhgkejhgk wrote: post #321
AuthAuth wrote: post #315

Earlier quoted context omitted.

There are certain domains where we can accept a high degree of mistakes and health is not one of them. People are reacting they way they are because its obvious that LLMs are currently not reliable enough to be trusted to distrubute health advice. To me it doesnt matter that ChatGPT health sometimes gives good advice or some people _feel_ like it helped them. I'm not sure I even trust people when they say that given…

I'm not saying LLMs don't make mistakes, I'm saying that some times get it right and so you shouldn't be surprised that you'll hear some positive stories. But you don't even believe these are success stories then, so I guess you can never be convinced.

You'll hear some positive stories from people who think rocks heal them

Re: ChatGPT Health is a marketplace, guess who is the product?

#323

Post by moi2388 »

well_ackshually wrote: post #126
simianwords wrote: post #85

Earlier quoted context omitted.

Do corporations use my google searches as data to hire me?

Do you have any proof they don't? Do you have any proof the "AI System" that they use to filter out candidates doesn't "accidentally" access data ? Are you willing to bet that Google, OpenAI, Anthropic, Meta, won't sell access to that information? Also, in some cases: they absolutely do. Try to get hired in Palantir and see how much they know about your browsing history. Anything related to national security or requi…

“ Do you have any proof they don't?”

Do you have any proof they don’t have a goose randomly deciding to hire you?

The lack of proof gives no credence to it actually happening

Re: ChatGPT Health is a marketplace, guess who is the product?

#324

Post by ekjhgkejhgk »

AuthAuth wrote: post #322
ekjhgkejhgk wrote: post #321

Earlier quoted context omitted.

I'm not saying LLMs don't make mistakes, I'm saying that some times get it right and so you shouldn't be surprised that you'll hear some positive stories. But you don't even believe these are success stories then, so I guess you can never be convinced.

You'll hear some positive stories from people who think rocks heal them

Yes, some people believe stupid things, so what?

It doesn't follow that it's a good idea to dismiss all input you disagree with.

But hey you do you.

Re: ChatGPT Health is a marketplace, guess who is the product?

#325

Post by avalys »

DBNO wrote: post #255
avalys wrote: post #141

Earlier quoted context omitted.

All we can do is share anecdotes here, but I have found ChatGPT to be confidently incorrect about important details in nearly every question I ask about a complex topic. Legal questions, question about AWS services, products I want to buy, the history a specific field, so many things. It gives answers that do a really good job of simulating what a person who knows the topic would say. But details are wrong everywhere…

I definitely agree that ChatGPT can be incorrect. I’ve seen that myself. In my experience, though, it’s more often right than wrong. So when you say “in nearly every question on complex topics", I’m curious what specific examples you’re seeing. Would you be open to sharing a concrete example? Specifically: the question you asked, the part of the answer you know is wrong, and what the correct answer should be. I have…

In one example, AWS has two options for automatic deletion of objects in S3 buckets that are versioned.

"Expire current versions" means that the object will be automatically deleted after some period.

"Permanently delete non-current versions" means that old revisions will be permanently removed after some period.

I asked ChatGPT for advice on configuring a bucket. Within a long list of other instructions, it said "Expire noncurrent versions after X days". In this case, such a setting does not exist, and the very similar "expire current versions" is exactly the wrong behavior. "Permanently delete noncurrent versions" is the option needed.

The prompt I used has other information in it that I don't want to share.

Re: ChatGPT Health is a marketplace, guess who is the product?

#326

Post by class3shock »

deaux wrote: post #264
class3shock wrote: post #236

Earlier quoted context omitted.

Or it's a placebo effect. And if it didn't work out and made you worse or, god forbid, the advice caused you to get seriously injured, then what? ChatGPT won't take any responsibility. I have so many issues with our current health system but an alternative is not an unreliable search tool that takes no responsibility for the information it provides.

> And if it didn't work out and made you worse or, god forbid, the advice caused you to get seriously injured, then what? ChatGPT won't take any responsibility. Realistically in 99% of actual cases where this happens due to human medical advice, the humans too won't take any responsibility.

A doctor is medically and legally responsible for the guidance they give patients. Are there cases where they give bad advice and avoid taking responsibility? Of course, as is the case with lawyers, engineers, etc., but there are standards they must meet, laws they must follow, and most importantly, consequences for not doing so.

This has zero responsibility, not some theoretical, "it may try to shirk responsibility" or "many people report it not taking responsibility", it by default takes no responsibility.

Re: ChatGPT Health is a marketplace, guess who is the product?

#327

Post by kundan_s__r »

simianwords wrote: post #76
jmyeet wrote: post #50

Earlier quoted context omitted.

This kind of comment scares me because it's an example of people substituring professional advice for an LLM where LLMs are known to hallucinate or otherwise simply make stuff up. I see this all the time when I write queries and get the annoying Gemini AI snippet on a subject I know about and often I'll see the AI make provably and objectively false statements.

You are exaggerating. LLMs simply don’t hallucinate all that often, especially ChatGPT. I really hate comments such as yours because anyone who has used ChatGPT in these contexts would know that it is pretty accurate and safe. People also can generally be trusted to identify good from bad advice. They are smart like that. We should be encouraging thoughtful ChatGPT use instead of showing fake concern at each opportun…

Whether or not Hallucination “happens often” depends heavily on the task domain and how you define correctness. In a simple conversational question about general knowledge, an LLM might be right more often than not — but in complex domains like cloud config, compliance, law, or system design, even a single confidently wrong answer can be catastrophic.

The real risk isn’t frequency averaged across all use cases — it’s impact when it does occur. That’s why confidence alone isn’t a good proxy: models inherently generate fluent text whether they know the right answer or not.

A better way to think about it is: Does this output satisfy the contract you intended for your use case? If not, it’s unfit for production regardless of overall accuracy rates.

Re: ChatGPT Health is a marketplace, guess who is the product?

#328

Post by deaux »

class3shock wrote: post #326
deaux wrote: post #264

Earlier quoted context omitted.

> And if it didn't work out and made you worse or, god forbid, the advice caused you to get seriously injured, then what? ChatGPT won't take any responsibility. Realistically in 99% of actual cases where this happens due to human medical advice, the humans too won't take any responsibility.

A doctor is medically and legally responsible for the guidance they give patients. Are there cases where they give bad advice and avoid taking responsibility? Of course, as is the case with lawyers, engineers, etc., but there are standards they must meet, laws they must follow, and most importantly, consequences for not doing so. This has zero responsibility, not some theoretical, "it may try to shirk responsibility"…

I care about the real world, not about theory. In theory, 99% of bad advice by doctors doesn't have any consequences for them. And that's being extremely conservative, in reality it's more like 0.01%, where an actual death can be directly attributed.

Re: ChatGPT Health is a marketplace, guess who is the product?

#329

Post by duskdozer »

mattmanser wrote: post #276
hackitup7 wrote: post #260

Earlier quoted context omitted.

I've had a similar positive experience and I'm really surprised at the cynicism here. You have a system that is good at reading tons of literature and synthesizing it, which then applies basic logic. What exactly do the cynics think that doctors do? I don't use LLMs as the final say, but I do find them pretty useful as a positive filter / quick gut check.

Because we've all used LLMs. The make stuff up. Doctors do not make stuff up. They agree with you. Almost all the time. If you ask an AI whether you have in fact been infected by a werewolf bite, they're going to try and find a way to say yes.

>Doctors do not make stuff up.

Haha. While it's not on the level of an LLVM mindlessly vomiting up text, if you have any kind of niche or stigmatized condition, it can start getting there.

Re: ChatGPT Health is a marketplace, guess who is the product?

#330

Post by class3shock »

deaux wrote: post #328
class3shock wrote: post #326

Earlier quoted context omitted.

A doctor is medically and legally responsible for the guidance they give patients. Are there cases where they give bad advice and avoid taking responsibility? Of course, as is the case with lawyers, engineers, etc., but there are standards they must meet, laws they must follow, and most importantly, consequences for not doing so. This has zero responsibility, not some theoretical, "it may try to shirk responsibility"…

I care about the real world, not about theory. In theory, 99% of bad advice by doctors doesn't have any consequences for them. And that's being extremely conservative, in reality it's more like 0.01%, where an actual death can be directly attributed.

In the real world 100% of bad advice from ai doesn't have any consequences for the ai companies.
Post reply on HN