Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

Post reply on HN
Print view View original

211–220 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#211

Post by bonesss »

ansgri wrote: post #203
sys32768 wrote: post #22

Earlier quoted context omitted.

Because it's the Cloud and we're told the cloud is better and more secure. In truth the company forced our hand by pricing us out of the on-premise solution and will do that again with the other on-premise we use, which is set to sunset in five years or so.

Probably has more to do with responsibility outsourcing: if SaaS has security breach AND they tell in the contract that they’re secure, then you’re not responsible. Sure, there may be reputational damage for you, but it’s a gamble with good odds in most cases. Storing lots of legal data doesn’t seem to be one of these cases though.

I see profits and outsourcing.

Selling an on-premise service requires customer support, engineering, and duplication of effort if you’re pushing to the cloud as well. Then you get the temptations and lock in of cloud-only tooling and an army of certified consultant drones whose resumes really really need time on AWS-doc-solution-2035, so the on premise becomes a constant weight on management.

SaaS and the cloud is great for some things some of the time, but often you’re just staring at the marketing playbook of MS or Amazon come to life like a golem.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#212

Post by LunaSea »

eru wrote: post #206
altmanaltman wrote: post #202

Earlier quoted context omitted.

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

> it's almost as if there's more stuff we do than just write code.. Yes, but adding these common sense considerations is actually something LLMs can already do reasonably well.

If you explicitly request it which means you need to know about it.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#213

Post by pbhjpbhj »

marginalx wrote: post #75
venturecruelty wrote: post #62

Earlier quoted context omitted.

And jail time for executives who are responsible for data leaks.

Are you saying executives cannot make mistakes ever (ask because you didn't qualify your statement)?

Are you saying the OP was just a single error, effectively an executives typo.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#214

Post by BrenBarn »

abustamam wrote: post #192
BrenBarn wrote: post #170

I'm less and less sure that when a billion-dollar company screws up this bad, the right thing to do is privately disclose it and let them fix it. This kind of thing just allows companies to go on taking people's money without facing the consequences of their mistakes.

What would you suggest the right thing to do would be? Edit: I agree with you that we shouldnt let companies like this get away with what amounts to a slap on the wrist. But everything else seems irresponsible as well.

I guess if I imagine the ideal world, it would be that you report it to the authorities and they impose penalties on the offender that are large enough that the company winds up significantly worse off than if they had just grown more slowly. In other words the punishment for moving fast and breaking things needs to be bad enough to outweigh the gains of doing so.

In the current world, I dunno. I guess it depends on what the company is. If it's something like a hedge fund or a fossil fuel company I think I'd be fine with some kind of wikileaks-like avenue for exposing it in such a way that it results in the company being totally destroyed.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#215

Post by BrenBarn »

mvkel wrote: post #174
barbazoo wrote: post #171

Earlier quoted context omitted.

Does a disclosure like this absolve them of any responsibility? They still violated whatever user privacy act.

It does. Most privacy laws are based on time-from-discovery. If they immediately sprung into action at the moment they were informed and remediated the issue, they're in compliance.

Right, that's the problem. There need to be standards that govern what can ever be released to customers/the public in the first place. When violations of those are discovered, the penalties should be based on time from release, so the longer it was out in the wild, the greater the penalty.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#216

Post by MangoToupe »

eru wrote: post #206
altmanaltman wrote: post #202

Earlier quoted context omitted.

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

> it's almost as if there's more stuff we do than just write code.. Yes, but adding these common sense considerations is actually something LLMs can already do reasonably well.

Clearly not

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#217

Post by MangoToupe »

AbstractH24 wrote: post #209
darkwater wrote: post #208

Earlier quoted context omitted.

In 90% of the cases. And if you don't know how to spot that other 10%, you are still screwed, cause someone else will found that (and you don't even need to be an elite black hat to find it).

What’s to say a human would catch this 10% either?

The salary you pay them, typically

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#218

Post by gbacon »

0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

This is HN. We understood exactly what “exposed … confidential files” meant before reading your overly dramatic scenario. As overdone as it is, it’s not even realistic. A likely single mother is likely tiny potatoes in comparison to deep-pocketed legal firms or large corporations.

The story is an example of the market self-correcting, but out comes this “building code” hobby horse anyway. All a software “building code” will do is ossify certain current practices, not even necessarily the best ones. It will tilt the playing field in favor of large existing players and to the disadvantage of innovative startups.

The model fails to apply in multiple ways. Building physical buildings is a much simpler, much less complex process with many fewer degrees of freedom than building software. Local city workers inspecting by the local municipality’s code at least has clear jurisdiction because of where the physical fixed location is. Who will write the “building code”? Who will be the inspectors?

This is HN. Of all places, I’d expect to see this presented as an opportunity for new startups, not calls for slovenly bureaucracy and more coercion. The private market is perfectly capable of performing this function. E&O and professional liability insurers if they don’t already will be soon motivated after seeing lawsuits to demand regular pentests.

The reported incident is a great reminder of caveat emptor.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#219

Post by hansmayer »

For all the talk in the blog of how "super-professional" their team was (probably just a courtesy on the side of the author, I don´t think he believes his own words either)... I have noticed using AI to produce some kind of API -OR- use a 3rd party point with integration into frontend, is almost guaranteed to produce code in which the frontend either exposes the API secrets directly in the frontend code (literally injecting it into a variable as string), or if you ask it for authentication, it will build some half-built lazy solution which makes no sense. So I imagine their "super-professional" team built this with AI, blindly trusting, probably even allowing it to commit and merge changes itself because if you are not merging 10K LoC a day with all this "great" technology, what are you even doing, right? It is not super-professional to work effectively with a blindfold on, I´d argue.
Post reply on HN