Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Employees are feeding sensitive data to ChatGPT, raising security fears

darkreading.com

Post reply on HN
Print view View original

291–300 of 355 posts

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#291

Post by senko »

Foobar8568 wrote: post #268
osterbit2 wrote: post #132

Earlier quoted context omitted.

To anyone who may be pasting code along the lines of 'convert this sql table schema into a [pydantic model|JSON Schema]' where you're pasting in the text, just ask it instead to write you a [python|go|bash|...] function that reads in a text file and 'converts an sql table schema to output x' or whatever. Related/not-related--great pandas docs replacement is another great+safe use-case. Point is, for a meaningful subs…

At first I was impressed by how easy it was to reach a data model with chatgpt, then I laughed as I tried to tweak it and use it. I realized it didn't really have any model concepts and was just using its various KB. I am unsure if the so called AI can think in models but so far, not but still an impressive assisting tool if you take care of its limitations. Another point where it lacks is in logic, my daughter has a…

> At first I was impressed by how easy it was to reach a data model with chatgpt, then I laughed as I tried to tweak it and use it. I realized it didn't really have any model concepts and was just using its various KB.

> I am unsure if the so called AI can think in models but so far, not but still an impressive assisting tool if you take care of its limitations.

I don't know. I'm using it for exactly that ("here's a problem, come up with a data model") and it gives a great starting point.[0]

Not perfect, but after that it's easy to tweak it the old-fashioned way.

I find its data modelling capabilities (in the domain I'm using it for - API services) to be rougly on par with a mid-level developer (for a handwavy definition of "midlevel").

[0] https://apibakery.com/demo/ai/

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#292

Post by tempestn »

cuuupid wrote: post #5

We block ChatGPT, as do most federal contractors. I think it’s a horrible exploit waiting to happen: - there’s no way they’re manually scrubbing out sensitive data so its bound to spill out from the training data when prompting the model - OpenAI is openly storing all this data they’re collecting to the extent that they’ve had several leaks now where people can see others’ conversations and data. We are one step away…

> they’ve had several leaks now where people can see others’ conversations and data

Do you have a source for this? I know some people have claimed to see others' data, but I haven't seen any evidence that that's what's actually being seen, vs LLM hallucinations. OpenAI claims, and I can't imagine they're lying, that the training data is fixed and ends in 2021, so I don't see how it would be possible for user prompts to be leaking into output, absent a massive and very unlikely bug (compared to the much more likely AI hallucination explanation).

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#293

Post by raincole »

KirillPanov wrote: post #287
raincole wrote: post #130

Earlier quoted context omitted.

> you can't just tell people not to use it Uh, why can't you tell people not to use it...? If security is that important for your company, of course you can tell your employees which tools to use. A fun fact: in many areas of TSMC, smart phones are banned. No one says "you can't just tell people not to use smart phones."

> in many areas of TSMC, smart phones are banned This does not surprise me at all . What I want to know is how they enforce it. Unless they have something better than "fear of somebody seeing you using the smartphone", it isn't getting enforced. If they do have something better I want to know what.

Metal detector.

No, I'm not joking. One of my high-school classmates works as R&D there. They ask you to pass through a metal detector gate, take away your phone if found, then give your a company phone for emergency call only. It's that strict (at least for R&D. Probably not for management and others).

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#294

Post by bob1029 »

KirillPanov wrote: post #287
raincole wrote: post #130

Earlier quoted context omitted.

> you can't just tell people not to use it Uh, why can't you tell people not to use it...? If security is that important for your company, of course you can tell your employees which tools to use. A fun fact: in many areas of TSMC, smart phones are banned. No one says "you can't just tell people not to use smart phones."

> in many areas of TSMC, smart phones are banned This does not surprise me at all . What I want to know is how they enforce it. Unless they have something better than "fear of somebody seeing you using the smartphone", it isn't getting enforced. If they do have something better I want to know what.

At Samsung, you typically walk through multiple sets of metal detectors and security before you can actually get into the fab. Anyone working in an office area can have a phone but they do some... stuff to it.

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#295

Post by taberiand »

Foobar8568 wrote: post #268
osterbit2 wrote: post #132

Earlier quoted context omitted.

To anyone who may be pasting code along the lines of 'convert this sql table schema into a [pydantic model|JSON Schema]' where you're pasting in the text, just ask it instead to write you a [python|go|bash|...] function that reads in a text file and 'converts an sql table schema to output x' or whatever. Related/not-related--great pandas docs replacement is another great+safe use-case. Point is, for a meaningful subs…

At first I was impressed by how easy it was to reach a data model with chatgpt, then I laughed as I tried to tweak it and use it. I realized it didn't really have any model concepts and was just using its various KB. I am unsure if the so called AI can think in models but so far, not but still an impressive assisting tool if you take care of its limitations. Another point where it lacks is in logic, my daughter has a…

Did you prime it before asking, so it was answering in the appropriate context?

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#296

Post by bob1029 »

somenameforme wrote: post #267
a_bonobo wrote: post #206

Earlier quoted context omitted.

llama's 7b model internally is, for me, on a totally different level quality-wise. Even when explicitly instructed to not make up stuff and just say 'I don't know', it will still go ahead and ramble and invent things. When I tell it to only use the prompt data it will still invent, or just ignore the prompt data. It's not useful for production (i.e., to be exposed to 'regular' non AI users). ChatGPT, on the other han…

One problem with the current way these models are being trained is that they have no idea of what they're saying. It's just a recursive guess the next word type algorithm. I would not expect the confidence levels of any given fragment, let alone an average, to be a meaningful predictor of truth. Also completely spitballing, I expect that a big chunk of OpenAI's 'secret sauce' is simple processing layers above and bey…

This is a very helpful observation.

I've been debating the idea of building tiers or layers of models to accomplish the same.

It very well could be that this go/no-go pre-processor is simply another ML model trained on a binary classification task. Stack a few of these and you can wind up with some interesting programming models.

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#297

Post by OJFord »

gwright wrote: post #207
ChatPGT wrote: post #164

Earlier quoted context omitted.

I'm doing that since day one. I can't believe people are pasting real data into this corporate black boxes.

What about Google Docs, Office 365, Github, AWS, Azure, Google Cloud, JIRA, Zendesk, etc? What is different about ChatGPT (if anything)?

I wouldn't put sensitive work data/employer IP in a personal Google Doc (et al.) either, no?

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#299

Post by a_bonobo »

bob1029 wrote: post #296
somenameforme wrote: post #267

Earlier quoted context omitted.

One problem with the current way these models are being trained is that they have no idea of what they're saying. It's just a recursive guess the next word type algorithm. I would not expect the confidence levels of any given fragment, let alone an average, to be a meaningful predictor of truth. Also completely spitballing, I expect that a big chunk of OpenAI's 'secret sauce' is simple processing layers above and bey…

This is a very helpful observation. I've been debating the idea of building tiers or layers of models to accomplish the same. It very well could be that this go/no-go pre-processor is simply another ML model trained on a binary classification task. Stack a few of these and you can wind up with some interesting programming models.

This would also explain the ease at which ChatGPT gets rid of escapes/bad prompts - they have an additional layer that assesses whether the question could be, for example, racist, and then spits out a 'Sorry as a language-model I am not trained to answer this kind of question'. No need to retrain the main 14B transformer model.

Re: Employees are feeding sensitive data to ChatGPT, raising security fears

#300

Post by nfm »

tempestn wrote: post #292
cuuupid wrote: post #5

We block ChatGPT, as do most federal contractors. I think it’s a horrible exploit waiting to happen: - there’s no way they’re manually scrubbing out sensitive data so its bound to spill out from the training data when prompting the model - OpenAI is openly storing all this data they’re collecting to the extent that they’ve had several leaks now where people can see others’ conversations and data. We are one step away…

> they’ve had several leaks now where people can see others’ conversations and data Do you have a source for this? I know some people have claimed to see others' data, but I haven't seen any evidence that that's what's actually being seen, vs LLM hallucinations. OpenAI claims, and I can't imagine they're lying, that the training data is fixed and ends in 2021, so I don't see how it would be possible for user prompts…

Yep: https://openai.com/blog/march-20-chatgpt-outage

Some kind of concurrency bug in a library they were using to retrieve cached data from Redis led to this leak.

> We took ChatGPT offline earlier this week due to a bug in an open-source library which allowed some users to see titles from another active user’s chat history. It’s also possible that the first message of a newly-created conversation was visible in someone else’s chat history if both users were active around the same time.

Post reply on HN