To safely deploy generative AI in health care, models must be open source • Hacker News Board

Re: To safely deploy generative AI in health care, models must be open source

#31

Post by KaiserPro » Fri, Dec 01, 2023, 8:22 AM UTC

Davidzheng wrote: post #13

KaiserPro wrote: post #11

I mean opensource is nice, but that's not actually going to make healthcare safer. Whatever flavour of AI needs to be deterministic, which llama, et al are not. even if you turn the temperature right down. As others have pointed out, its the training set that actually makes a model behave, hence why models are freely given away by large companies.

Why must it be deterministic? Humans are not but we're still can have trust in humans.

Because if you are trying to assess the safety of some process and there is a 0.5% chance that it will give you deadly results, its not good.

Moreover, if its not deterministic, how can you assess if its safe? Sure you can run many many iterations, but how do you know when its safe enough? LLMs encourage freeform entry, which means the testing space is fucking massive.

Does writing with a different syntactic style give different outcomes? Does spelling mistakes lead to increase morbidity? thats a test plan I don't want to have to run (unless you are paying me megabucks.)

Humans are not deterministic as you point out, which is why you need to control for as many unknowns as possible when testing in a health setting.

Re: To safely deploy generative AI in health care, models must be open source

#32

Post by blitzar » Fri, Dec 01, 2023, 10:38 AM UTC

TaylorAlexander wrote: post #3

Recently there has been a trend in calling models with weights and code available "open source" even if the training data is not available. For safe deployment in health care and other safety critical fields, transparency on the training data and process are vital too, which means developing clear terminology for models full transparency! Even this article title suffers from this ambiguity.

Eventually it will become the norm (as it is around these parts) for the Ai to provide sources for its wild claims.

If the answer is "Neck bone connected to the head bone" im going to want to see a source that isnt Dem Bones

Re: To safely deploy generative AI in health care, models must be open source

#33

Post by camkego » Fri, Dec 01, 2023, 11:28 AM UTC

blitzar wrote: post #32

TaylorAlexander wrote: post #3

Recently there has been a trend in calling models with weights and code available "open source" even if the training data is not available. For safe deployment in health care and other safety critical fields, transparency on the training data and process are vital too, which means developing clear terminology for models full transparency! Even this article title suffers from this ambiguity.

Eventually it will become the norm (as it is around these parts) for the Ai to provide sources for its wild claims. If the answer is "Neck bone connected to the head bone" im going to want to see a source that isnt Dem Bones

There is still a large difference between an AI being able to cite it's claims, and having the original training data and exact code and process in which to convert the training data into the weights used in inference.

If you cannot re-create the weights and model used for inference, a release's value is somewhat limited vs releases where the inference model can be re-created. (It's kind of like the limited value of scientific papers where the results cannot be reproduced due to a lack of detail)

Re: To safely deploy generative AI in health care, models must be open source

#34

Post by lrvick » Fri, Dec 01, 2023, 2:06 PM UTC

dontreact wrote: post #29

lrvick wrote: post #17

Earlier quoted context omitted.

Redhat, Element, Prusa, Adafruit, Sidero Labs, Arduino... plenty of companies that open source everything or almost everything and have have staying power. Many consumers, myself included, will -only- pay for technology if it is open source. In fact if something is proprietary I feel I am being cheated anyway and I might as well pirate it until I find something open to support. Many of us are willing to pay for time…

I looked through a bit of the companies here and I don’t see any companies that have to retain a quality system and staff to stand behind their products permanently. These models seem to work better when you can just put stuff out there and occasionally pop in to help? This is not a model that can work with regulated medical products. There is a very significant cost to maintaining static artifacts and I don’t see ho…

You must have overlooked RedHat and Sidero Labs.

How many hospitals download the linux kernel and manage their own servers?

I would assume very few. They likely lack the expertise, and pay third party companies to manage the servers, patches, and updates for them... even when the software is open source.

Re: To safely deploy generative AI in health care, models must be open source

#35

Post by runnerup » Sat, Dec 02, 2023, 6:13 PM UTC

Spivak wrote: post #8

troupo wrote: post #6

Earlier quoted context omitted.

That's why EU's upcoming AI regulation requires foundational models to have full documentation , including detailed descriptions of training data etc.

I can't fathom why they didn't just require the models to make available the training data itself. Sure you might need to fork some cash so they can ship you hard drives but surely being audited by someone anyone is better than none.

Training data for a medical diagnosis model would likely include enough info to de-anonymize the info for some participants (age, sex, zip code, descriptions). I'm not sure what the answer should be but I'm uncomfortable with the medical training data being provided freely to the world.

Re: To safely deploy generative AI in health care, models must be open source

#36

Post by dontreact » Wed, Dec 06, 2023, 2:30 AM UTC

lrvick wrote: post #34

dontreact wrote: post #29

Earlier quoted context omitted.

I looked through a bit of the companies here and I don’t see any companies that have to retain a quality system and staff to stand behind their products permanently. These models seem to work better when you can just put stuff out there and occasionally pop in to help? This is not a model that can work with regulated medical products. There is a very significant cost to maintaining static artifacts and I don’t see ho…

You must have overlooked RedHat and Sidero Labs. How many hospitals download the linux kernel and manage their own servers? I would assume very few. They likely lack the expertise, and pay third party companies to manage the servers, patches, and updates for them... even when the software is open source.

If installing these things required passing FDA trials before you could even do the first install, it’s hard to imagine anyone self funding the regulatory trial and then open sourcing the result.

These businesses you mention are different because you can start getting paid as soon as you build some expertise in how to operationalize the systems.