Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

111–120 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#111

Earlier quoted context omitted.

Presumably they were paid for finding the bug and inn accepting relinquished their right to blog about it.

No, you relinquish the right when you agree to their TOS irrespective of if they pay you.

TOS != law

They will stop letting you use the service. That's the recourse for breaking the TOS.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#112
My thing is, even ingesting the BOK should have been done in phases, to avoid having all your virtual eggs in one basket or nest at any ONE time. Staggering tokens to these compartments would not have cost them anything at all . I always say, whatever convenience you enjoy yourself, will be highly appreciated by bad actors... WHEN, not if.. they get thru.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#113

> ... after looking through minified code, which SUCKS to do ... AI tends to be good at un-minifying code.

Legit question: when working on finding security issues, are there any guidelines on what you can send to LLMs/AI?

Personally, I'd just use common sense and good judgment. At the end of the day, would you want someone to hand your address, and other private data to OpenAI just like that? Probably not. So don't paste customer data into it if you can avoid it.

On the other hand, minified code is literally published by the company. Everyone can see it and do with it as they please. So handing that over to an AI to un-minify is not really your problem, since you're not the developer working on the tool internally.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#114
post #8
post #4

[flagged]

It's a little hilarious. First, as an organization, do all this cybersecurity theatre, and then create an MCP/LLM wormhole that bypasses it all. All because non-technical folks wave their hands about AI and not understanding the most fundamental reality about LLM software being fundamentally so different than all the software before it that it becomes an unavoidable black hole. I'm also a little pleased I used two sp…

Assuming a 101 security program past the quality bar, there are a number of reason why this can still happen at companies.

Summarized as - security is about risk acceptance, not removal. There’s massive business pressure to risk accept AI. Risk acceptance usually means some sort of supplemental control that’s not the ideal but manages. There are very little of these with AI tools however - small vendors, they’re not really service accounts but IMO best way to monitor them probably is that, integrations are easy, eng companies hate devs losing admin of some kind but if you have that random AI on endpoints becomes very likely.

I’m ignoring a lot of nuance but solid sec program blown open by LLM vendors is going to be common, let alone bad sec programs. Many sec teams I think are just waiting for the other shoe to drop for some evidentiary support while managing heavy pressure to go full bore AI integration until then.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#116
post #38
post #35

Earlier quoted context omitted.

Not every organization prioritizes being able to ship a code change at the drop of a hat. This often requires organizational dedication to heavy automated testing a CI, which small companies often aren't set up to do.

I can't believe that any company takes a month to ship something. Even if they don't have CI, surely they'd prefer to break the app (maybe even completely) than risk all their legal documents exfiltrated.

It’d be pretty reasonable to take the whole API down in this scenario, and put it back up once it’s patched. They’d lose tons of cash but avoid being liable for extreme amounts of damages.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#117
post #95

> November 20, 2025: I followed up to confirm the patch was in place from my end, and informed them of my intention to write a technical blog post. Can that company tell you to cease and desist? How does the law work?

Lawyers can and will send cease and desist letters to people whether or not there is any legal basis for it. Often the threat of a lawsuit, even a meritless one, is enough to keep people quiet.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#118
post #12

I'm always a bit surprised how long it can take to triage and fix these pretty glaring security vulnerabilities. October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed. Sure the actual bug ended up being (what I imagine to be) a Is the issue that people aren't checking their security@ email addresses? People are on holiday? These emai…

Another aspect to consider: when you reduce the amount of permission anything has (like here the returned token), you risk breaking something.

In a complex system it can be very hard to understand what will break, if anything. In a less complex system, it can still be hard to understand if the person who knows the security model very well isn't available.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#119

I work for a finance firm and everyone is wondering why we can store reams of client data with SaaS Company X, but not upload a trust document or tax return to AI SaaS Company Y. My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it. This article demonstrates that, but it does sort of beg the question…

Does SaaS X/Cloud offer IAM capabilities? Or going further, do they dogfood their own access via the identity and access policies? If so, and you construct your own access policy, you have relative peace of mind.

If SaaS Y just says "Give me your data and it will be secure", that's where it gets suspect.

Post reply on HN