Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

Post reply on HN
Print view View original

281–290 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#281

Post by agos »

altmanaltman wrote: post #202
0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

See also: HN told me that regulation is bad and this is why the EU is behind!

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#282

Post by jimbokun »

j45 wrote: post #277
jimbokun wrote: post #67

Earlier quoted context omitted.

My first reaction to the announcement of MCP was that I must be missing something. Surely giving an LLM unlimited access to protected data is going to introduce security holes?

Agree. It’s assuming and estimating it will behave like other software before it when it’s nothing like the software that came before it. LLMs today won’t behave like the software we’re used to where 1+1 will equal 2 every time.

It’s like handing access to your sensitive data to a complete stranger and hoping for the best.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#283

Post by AbstractH24 »

MangoToupe wrote: post #276
AbstractH24 wrote: post #263

Earlier quoted context omitted.

Salaries make humans infallible?

No, but it makes them motivated to be thorough. There is no way to motivate a chatbot (to do better or to any end).

But money is a way to motivate the people who created AI to create better AI. Because if it doesn't perform as expected, either people won't use it or they'll turn to a competitor next time they need to do something. And these companies need recurring revenue.

If we're saying the way to ensure competency is to instill fear of not getting money tomorrow as a consequence of failure, then AI companies and humans are on equal footing.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#284

Post by joshribakoff »

ethbr1 wrote: post #275
joshribakoff wrote: post #268

Earlier quoted context omitted.

You may want to try using it, anecdotes often differ from theories, especially when they are being sold to you for profit. It takes maybe a few days to see a pattern of ignoring simple instructions even when context is clean. Or one prompt fixes one issue and causes new issues, rinse and repeat. It requires human guidance in practice.

Strongman: LLMs aren't a tool, they're fuzzy automation. And what keeps security problems from making it into prod in the real world? Code review, testing, static and dynamic code scanning, and fuzzing. Why aren't these things done? Because there isn't enough people-time and expertise. So in order for LLMs to improve security, they need to be able to improve our ability to do one of: code review, testing, static and…

There is an argument here that the LLM is a tool that can multiply the addition or removal of the defects depending on how it is wielded.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#285

Post by ethbr1 »

joshribakoff wrote: post #284
ethbr1 wrote: post #275

Earlier quoted context omitted.

Strongman: LLMs aren't a tool, they're fuzzy automation. And what keeps security problems from making it into prod in the real world? Code review, testing, static and dynamic code scanning, and fuzzing. Why aren't these things done? Because there isn't enough people-time and expertise. So in order for LLMs to improve security, they need to be able to improve our ability to do one of: code review, testing, static and…

There is an argument here that the LLM is a tool that can multiply the addition or removal of the defects depending on how it is wielded.

I think the father figure of a developer who was bitten by a radioactive spider once made a similar quip.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#286

Post by eru »

MangoToupe wrote: post #276
AbstractH24 wrote: post #263

Earlier quoted context omitted.

Salaries make humans infallible?

No, but it makes them motivated to be thorough. There is no way to motivate a chatbot (to do better or to any end).

You can run multiple chatbots in parallel. Use different models and different setups.

It's like having multiple people audit your systems. Even if everyone only catches 90%, as long as they don't catch exactly the same 90%, this parallel effort helps.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#287

Post by eru »

delaminator wrote: post #199
theoldgreybeard wrote: post #163

Earlier quoted context omitted.

Maybe I have just been lucky, but I have not had the displeasure of working with people either tha incompetent or willfully ignorant yet.

I’m governed by them Reeves orders Treasury inquiry over Budget leaks Chancellor’s policies found their way to the press before she announced them to MPs https://www.telegraph.co.uk/news/2025/12/03/reeves-orders-tr...

You might want to consider voting with your feet?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#288

Post by eru »

chanux wrote: post #266
stingraycharles wrote: post #165

Earlier quoted context omitted.

SOC2 is mainly to check boxes, and forces you to think about a few things. There’s no real / actual audit, and in my experience the pen tests are very much a money grab. You’re paying way too much money for some “pentesting” automated suite to run. The auditors themselves pretty much only care that you answered all questions, they don’t really care what the answers are and absolutely aren’t going to dig any deeper. (…

Once this type of issue gets publicized, does that in anyway affect the certification?

Sometimes scandals affect these things. But it's hard to predict.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#289

Post by eru »

aitchnyu wrote: post #195
habosa wrote: post #150

They took a month to fix this? That’s beyond inexcusable. I can’t imagine how any customer could justify working with them going forward. Also … shows you what a SOC 2 audit is worth: https://www.filevine.com/news/filevine-proves-industry-leade... Even the most basic pentest would have caught this.

Is there any stricter standard? Should one strive for PCI-DSS even if they are a regular SaaS?

Whatever Google does internally would be a much stricter standard, but I'm not sure they've written it up for outsiders to use, alas.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#290

Post by rendaw »

altmanaltman wrote: post #202
0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

AFAICT Filevine doesn't use AI programming: https://www.filevine.com/jobs/d64dfff5-e36f-4db6-adac-0fc082... There's no mention of AI there other than writing code to integrate AI pipelines.

I've seen a lot of job ads (Canva) lately that mandate AI use or AI experience, and as an AI company if they wanted that I think they would have put it in the ad.

For the record I think I may be fine with the insincerity of selling AI but not using it!

Post reply on HN