Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

171–180 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#171

I'm less and less sure that when a billion-dollar company screws up this bad, the right thing to do is privately disclose it and let them fix it. This kind of thing just allows companies to go on taking people's money without facing the consequences of their mistakes.

Does a disclosure like this absolve them of any responsibility? They still violated whatever user privacy act.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#172

The bigwigs at my company want to build out a document management suite. After talking to VP of technology about requirements I ask about security as well as what the regulatory requirements are and all I get is a blank stare. I used to think developers had to be supremely incompetent to end up with vulnerabilities like this. But now I understand it’s not the developers who are incompetent…

I've had the same. Ask them to come up with a ToS and they're like "we'll talk about that in an upcoming meeting" it's been a few years now with nothing.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#173
So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents. This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor.

Imagine the potential impact. You're a single mother, fighting for custody of your kids. Your lawyer has some documentation of something that happened to you, that wasn't your fault, but would look bad if brought up in court. Suddenly you receive a phone call - it's a mysterious voice, demanding $10,000 or they will send the documents to the opposition. Neither of them knows each other; someone just found a trove of documents in an open back door and wanted to make a quick buck.

This is exactly what a software building code would address (if we had one!). Just like you can't open a new storefront in a new building without it being inspected, you should not be able to process millions of sensitive files without having your software's building inspected. The safety and privacy of all of us shouldn't be optional.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#174

I'm less and less sure that when a billion-dollar company screws up this bad, the right thing to do is privately disclose it and let them fix it. This kind of thing just allows companies to go on taking people's money without facing the consequences of their mistakes.

Does a disclosure like this absolve them of any responsibility? They still violated whatever user privacy act.

It does. Most privacy laws are based on time-from-discovery. If they immediately sprung into action at the moment they were informed and remediated the issue, they're in compliance.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#175
post #174

Earlier quoted context omitted.

Does a disclosure like this absolve them of any responsibility? They still violated whatever user privacy act.

It does. Most privacy laws are based on time-from-discovery. If they immediately sprung into action at the moment they were informed and remediated the issue, they're in compliance.

So is that true if they find out when the public does too? It seems that disclosing it privately has some upside (protecting the users) and no downside.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#176

[flagged]

Please don't do this here. If a comment seems unfit for HN, please flag it and email us at hn@ycombinator.com so we can have a look.

We detached this subthread from https://news.ycombinator.com/item?id=46137863 and marked it off topic.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#177

It's so great that they allowed him to publish a technical blog post. I once discovered a big vulnerability in a listed consumer tech company -- exposing users' private messages and also allowing to impersonate any user. The company didn't allow me to write a public blogpost.

Yet another reason these disclosures should be anonymous (from the reporting side).

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#178
post #174

Earlier quoted context omitted.

It does. Most privacy laws are based on time-from-discovery. If they immediately sprung into action at the moment they were informed and remediated the issue, they're in compliance.

So is that true if they find out when the public does too? It seems that disclosing it privately has some upside (protecting the users) and no downside.

That depends more on what the Privacy Policy is of the service, which you agree to when you sign up and use it

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#179

I've worked in several "agentic" roles this year alone (I'm very poachable lol) and otherwise well structured engineering orgs have lost their goddamn minds with move fast and break things because they're worried that OpenAI/Google/Meta/Amazon/Anthropic will release the tool they're working on tomorrow literally all of them are like this

Old school blue chip type of companies are like this too. They’ve thrown all the process and caution they used to have to the wind so that they can… apply AI to their IT org which isn’t even their core business?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#180
post #27
post #21

Earlier quoted context omitted.

security@ emails do get a lot of spam. It doesn't get talked about very much unless you're monitoring one yourself, but there's a fairly constant stream of people begging for bug bounty money for things like the Secure flag not being set on a cookie. That said, in my experience this spam is still a few emails a day at the most, I don't think there's any excuse for not immediately patching something like that. I guess…

This. There is so much spam from random people about meaningless issues in our docs. AI has made the problem worse. Determining the meaningful from the meaningless is a full time job.

Training a tech support team of interns to solve all of them would be an enviable hacker or software dev training program.
Post reply on HN