What's the feedback loop here for ChatGPT?
For it to get better, it needs to know outcomes of its diagnosis.
Are people just typing back to ChatGPT saying "you're wrong / you're right"?
I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
191–200 of 240 posts
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#192[dead]
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#193Health metrics are absolutely tarnished by a lack of proper context. Unsurprisingly, it turns out that you can't reliably take a concept as broad as health and reduce it to a number. We see the same arguments over and over with body fat percentages, vo2 max estimates, BMI, lactate thresholds, resting heart rate, HRV, and more. These are all useful metrics, but it's important to consider them in the proper context tha…
[dead]
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#194> Despite having access to my weight, blood pressure and cholesterol, ChatGPT based much of its negative assessment on an Apple Watch measurement known as VO2 max, the maximum amount of oxygen your body can consume during exercise. Apple says it collects an “estimate” of VO2 max, but the real thing requires a treadmill and a mask. Apple says its cardio fitness measures have been validated, but independent researchers…
> I think the blame more rests on Apple for falsely representing the quality of their product There was plenty of other concerning stuff in that article. And from a quick read it wasn't suggested or implied the VO2 max issue was the deciding factor for the original F score the author received. The article did suggest many times over the ChatGPT is really not equipped for the task of health diagnosis. > There was anot…
> There was plenty of other concerning stuff in that article.
Yeah for sure, I probably didn't make it clear enough but I do fault OpenAI for this as much as or maybe more than Apple. I didn't think that needed to be stressed since the article is already blasting them for it and I don't disagree with most of that criticism of OpenAI.
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#195> Despite having access to my weight, blood pressure and cholesterol, ChatGPT based much of its negative assessment on an Apple Watch measurement known as VO2 max, the maximum amount of oxygen your body can consume during exercise. Apple says it collects an “estimate” of VO2 max, but the real thing requires a treadmill and a mask. Apple says its cardio fitness measures have been validated, but independent researchers…
Well if it doesn't know the quality of the data and especially if it would be dangerous to guess then it should probably say it doesn't have an answer.
I don't disagree, but that reinforces my point above I think. If AI has to assume the data is of poor quality, then there's no point in even trying to analyze it. The options are basically:
1. Trust the source of the data to be honest about it's quality
Or
2. Distrust the source
Approach number 2 basically means we can never do any analysis on it.
Personally I'd rather have a product that might be wrong than none at all, but that's a personal preference.
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#196> Despite having access to my weight, blood pressure and cholesterol, ChatGPT based much of its negative assessment on an Apple Watch measurement known as VO2 max, the maximum amount of oxygen your body can consume during exercise. Apple says it collects an “estimate” of VO2 max, but the real thing requires a treadmill and a mask. Apple says its cardio fitness measures have been validated, but independent researchers…
> Should it always assume bad data or potentially bad data? If so, that seems like it would defeat the point of having data at all as you could never draw any conclusions from it. Yes. You, and every other reasoning system, should always challenge the data and assume it’s biased at a minimum. This is better described as “critical thinking” in its formal form. You could also call it skepticism. That impossibility of d…
I mostly agree with you, but I think it's important to consider what you're doing with the data. If we're doing rigorous science, or making life-or-death decisions on it, I would 100% agree. But if we're an AI chatbot trying to offer some insight, with a big disclaimer that "these results might be wrong, talk to your doctor" then I think that's quite overkill. The end result would be no (potential) insight at all and no chance for ever improving since we'll likely never get a to a point where we could fully trust the data. Not even the best medical labs are always perfect.
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#197Earlier quoted context omitted.
How so?
ChatGPT will actually look at your whole medical history, listen to you, think and check multiple different options before making a decision. You can spend hours chatting with it back and forth. An average human doctor has maybe 15 minutes allotted to getting to know you, analyse and determine a course of action. Which is usually "take some ibuprofen and let's see if it goes away". Then you go again in two weeks with…
> has maybe 15 minutes allotted to getting to know you [...] Then you go again in two weeks with the same thing, it's a different doctor and the context has been reset
This is not how doctors work in most of the world. Not having an actual primary care physician that is able to keep track of each patient over multiple years means they are skipping out on one of their most important duties. You should advocate for a better standard of care rather than resorting to hallucinating chatbots.
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#198Earlier quoted context omitted.
> The most notorious of which is BMI; it is practically a category error to infer someone's health or risk by individual BMI, and yet doing so remains widespread amongst people that are supposed to know better. BMI works fine for people who aren't very muscular, which is the great majority of people. Waist to height ratio might be more informative for people with higher muscle mass.
As a person who has been told I'm "morbidly obese" for decades now, I will say that doctors at almost every level look at your chart not you. I've been told time and time again that until I get my weight under control, my health will suffer. I'm 5'8" and weigh on average 210lbs. My BMI isn't even morbidly obese, it is 31, which is just "regular" obese, but on top of that, a DEXA scan shows that I am actually only 25%…
I'd consider 5'8 and 210lbs morbidly obese. An average male at 5'8 should generally weigh about 150lbs and no more than 164lbs.
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#199LLMs are not a mythical universal machine learning model that you can feed any input and have it magically do the same thing a specialized ML model could do. You can't feed an LLM years of time-series meteorological data, and expect it to work as a specialized weather model, you can't feed it years of medical time-series and expect it to work as a model specifically trained, and validated on this specific kind of dat…
You state a lot of things without testing it first? A LLM has structures in its latent space which allows it to do basic math, it has also seen enough data that it has probably structures in it to detect basic trends. A LLM doesn't just generate a stream of tokens. It generates an embedding and searches/does something in its latent space, then returns tokens. And you don't even know at all what LLM Interfaces do in t…
That's not how it works.
Re: I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
#200I dunno, if the Apple Watch said he had a vo2max of 30, that probably means he can’t run a mile in less than 12 minutes or so. He’s probably not at all healthy…
I had a "below average" VO2 max score based on my Apple Watch measurements. It was ~40 mL/kg/min, in the span of about a month it jumped up to 53 mL/kg/min, which is "high" for my age group. So what happened? I started running instead of cycling as my primary form of cardio. My hypothesis is that the apple watch estimates higher if you are running rather than pedaling. I definitely don't think my cardio vascular went…
Yeah I just ignore it, when I was biking 40+ miles a week this summer it says my VO2 max was 18, which is just absurd. Maybe because my arm is really hairy I don’t know.