Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

Post reply on HN
Print view View original

201–210 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#201

Post by vkou »

eru wrote: post #164
theoldgreybeard wrote: post #163

Earlier quoted context omitted.

Maybe I have just been lucky, but I have not had the displeasure of working with people either tha incompetent or willfully ignorant yet.

Oh, I should have been more careful in my formulation: There are organisations that are generally competent, and there are places that are less competent. It's not all that uncommon for the whole organisation to be generally incompetent. The saddest places (for me) are those where almost every individual you talk to seems generally competent, but judging by their output the company might as well be stuffed by idiots.…

> Something in the way they are organised suppresses the competence.

It's a natural outcome of authoritarian structures when the people at the top are idiots. When that happens, the whole organization rots.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#202

Post by altmanaltman »

0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#203

Post by ansgri »

sys32768 wrote: post #22
layer8 wrote: post #20

Earlier quoted context omitted.

The question is what reason did you have to trust SaaS Company X in the first place?

Because it's the Cloud and we're told the cloud is better and more secure. In truth the company forced our hand by pricing us out of the on-premise solution and will do that again with the other on-premise we use, which is set to sunset in five years or so.

Probably has more to do with responsibility outsourcing: if SaaS has security breach AND they tell in the contract that they’re secure, then you’re not responsible. Sure, there may be reputational damage for you, but it’s a gamble with good odds in most cases.

Storing lots of legal data doesn’t seem to be one of these cases though.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#204

Post by balamatom »

Aurornis wrote: post #137
balamatom wrote: post #82

Earlier quoted context omitted.

Or maybe these are people who learned from a LLM that English is supposed to sound like this if you want to be permitted to communicate a.k.a. "to be taken into consideration"! Which is wrong and also kinda sucks, but also it sucks and is wrong for a kinda non-obvious reason. Or, bear with me there, maybe things aren't so far downhill yet, these users just learned how English is supposed to sound, from the same place…

Or maybe the 2 month old account posting repetitive comments and using the exact patterns common to AI generated comment is, actually, posting LLM generated content. > So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere. FYI, spammers love LLM generated posting because it allows them to "season" accounts on sites like Hacker News and Reddit without much effort. Post enough plausib…

Wasn't there some sort of escape hatch for situations like that - for when it becomes impossible to trust the agora?

It would be massively funny if that escape hatch just sort of disappeared while we were looking at something else.

Your point stands, though.

>exact patterns common to AI generated comment

How can there be exact patterns to it?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#206

Post by eru »

altmanaltman wrote: post #202
0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

> it's almost as if there's more stuff we do than just write code..

Yes, but adding these common sense considerations is actually something LLMs can already do reasonably well.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#207

Post by samdung »

eru wrote: post #153
theoldgreybeard wrote: post #146

The bigwigs at my company want to build out a document management suite. After talking to VP of technology about requirements I ask about security as well as what the regulatory requirements are and all I get is a blank stare. I used to think developers had to be supremely incompetent to end up with vulnerabilities like this. But now I understand it’s not the developers who are incompetent…

There's enough incompetence at all levels to go around.

Incompetence compounds at an astonishing rate.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#208

Post by darkwater »

eru wrote: post #206
altmanaltman wrote: post #202

Earlier quoted context omitted.

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

> it's almost as if there's more stuff we do than just write code.. Yes, but adding these common sense considerations is actually something LLMs can already do reasonably well.

In 90% of the cases. And if you don't know how to spot that other 10%, you are still screwed, cause someone else will found that (and you don't even need to be an elite black hat to find it).

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#209

Post by AbstractH24 »

darkwater wrote: post #208
eru wrote: post #206

Earlier quoted context omitted.

> it's almost as if there's more stuff we do than just write code.. Yes, but adding these common sense considerations is actually something LLMs can already do reasonably well.

In 90% of the cases. And if you don't know how to spot that other 10%, you are still screwed, cause someone else will found that (and you don't even need to be an elite black hat to find it).

What’s to say a human would catch this 10% either?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#210

Post by mrweasel »

habosa wrote: post #150

They took a month to fix this? That’s beyond inexcusable. I can’t imagine how any customer could justify working with them going forward. Also … shows you what a SOC 2 audit is worth: https://www.filevine.com/news/filevine-proves-industry-leade... Even the most basic pentest would have caught this.

The time to fix isn't really important, assuming that they took the system offline in the mean time... but we all know they didn't, because that would cost to much.
Post reply on HN