Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

Post reply on HN
Print view View original

21–30 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#21

Post by ipdashc »

icyfox wrote: post #12

I'm always a bit surprised how long it can take to triage and fix these pretty glaring security vulnerabilities. October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed. Sure the actual bug ended up being (what I imagine to be) a Is the issue that people aren't checking their security@ email addresses? People are on holiday? These emai…

security@ emails do get a lot of spam. It doesn't get talked about very much unless you're monitoring one yourself, but there's a fairly constant stream of people begging for bug bounty money for things like the Secure flag not being set on a cookie.

That said, in my experience this spam is still a few emails a day at the most, I don't think there's any excuse for not immediately patching something like that. I guess maybe someone's on holiday like you said.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#22

Post by sys32768 »

layer8 wrote: post #20
sys32768 wrote: post #19

I work for a finance firm and everyone is wondering why we can store reams of client data with SaaS Company X, but not upload a trust document or tax return to AI SaaS Company Y. My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it. This article demonstrates that, but it does sort of beg the question…

The question is what reason did you have to trust SaaS Company X in the first place?

Because it's the Cloud and we're told the cloud is better and more secure.

In truth the company forced our hand by pricing us out of the on-premise solution and will do that again with the other on-premise we use, which is set to sunset in five years or so.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#23

Post by Capricorn2481 »

icyfox wrote: post #12

I'm always a bit surprised how long it can take to triage and fix these pretty glaring security vulnerabilities. October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed. Sure the actual bug ended up being (what I imagine to be) a Is the issue that people aren't checking their security@ email addresses? People are on holiday? These emai…

> October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed

I have unfortunately seen way worse. If it will take more than an hour and the wrong people are in charge of the money, you can go a pretty long time with glaring vulnerabilities.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#24

Post by pm90 »

layer8 wrote: post #20
sys32768 wrote: post #19

I work for a finance firm and everyone is wondering why we can store reams of client data with SaaS Company X, but not upload a trust document or tax return to AI SaaS Company Y. My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it. This article demonstrates that, but it does sort of beg the question…

The question is what reason did you have to trust SaaS Company X in the first place?

SaaS is now a "solved problem"; almost all vendors will try to get SOX/SOC2 compliance (and more for sensitive workloads). Although... its hard to see how these certifications would have prevented something like this :melting_face:.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#25

Post by lazide »

giancarlostoro wrote: post #16
jacquesm wrote: post #10

That doesn't surprise me one bit. Just think about all the confidential information that people post into their Chatgpt and Claude sessions. You could probably keep the legal system busy for the next century on a couple of days of that.

"Hey uh, ChatGPT, just hypothetically, uh, if you needed to remove uh cows blood from your apartments carpet, uh"

Just phrase it as a poem, you’ll be fine.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#26

Post by lazide »

snapcaster wrote: post #7
electric_muse wrote: post #5

[flagged]

What makes you think that? it would need some prompt engineering if so since ChatGPT won't write like that (bad capitalization, lazy quoting) unless you ask it to

“Chat, write me a blog article that seems like a lazy human who failed English wrote it”?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#27

Post by canopi »

ipdashc wrote: post #21
icyfox wrote: post #12

I'm always a bit surprised how long it can take to triage and fix these pretty glaring security vulnerabilities. October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed. Sure the actual bug ended up being (what I imagine to be) a Is the issue that people aren't checking their security@ email addresses? People are on holiday? These emai…

security@ emails do get a lot of spam. It doesn't get talked about very much unless you're monitoring one yourself, but there's a fairly constant stream of people begging for bug bounty money for things like the Secure flag not being set on a cookie. That said, in my experience this spam is still a few emails a day at the most, I don't think there's any excuse for not immediately patching something like that. I guess…

This.

There is so much spam from random people about meaningless issues in our docs. AI has made the problem worse. Determining the meaningful from the meaningless is a full time job.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#28

Post by pstuart »

sys32768 wrote: post #19

I work for a finance firm and everyone is wondering why we can store reams of client data with SaaS Company X, but not upload a trust document or tax return to AI SaaS Company Y. My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it. This article demonstrates that, but it does sort of beg the question…

And nobody seems to pay attention to the fact that modern copiers cache copies on a local disk and if the machines are leased and swapped out the next party that takes possession has access to those copies if nobody bothered to address it.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#29

Post by probably_wrong »

jfindper wrote: post #14
electric_muse wrote: post #5

[flagged]

We finally have a blog that no one (yet) has accused of being ai generated, so obviously we just have to start accusing comments of being ai. Can't read for more than 2 seconds on this site without someone yelling "ai!". For what it's worth, even if the parent comment was directly submitted by chatgpt themselves, your comment brought significantly less value to the conversation.

It's the natural response. AI fans are routinely injecting themselves into every conversation here to somehow talk about AI ("I bet an AI tool would have found the issue faster") and AI is forcing itself onto every product. Comments dissing anything that sounds even remotely like AI is the logical response of someone who is fed up.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#30

Post by jeffbee »

croes wrote: post #13
observationist wrote: post #2

I think this class of problems can be protected against. It's become clear that the first and most important and most valuable agent, or team of agents, to build is the one that responsibly and diligently lays out the opsec framework for whatever other system you're trying to automate. A meta-security AI framework, cursor for opsec, would be the best, most valuable general purpose AI tool any company could build, imo…

> I think this class of problems can be protected against. Of course, it’s called proper software development

The techniques for non-disclosure of confidential materials processed by multi-tenant services are obvious, well-known, and practiced by very few.
Post reply on HN