Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

Post reply on HN
Print view View original

271–280 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#271

Post by joshribakoff »

windexh8er wrote: post #262
eric-burel wrote: post #251

Earlier quoted context omitted.

You may want to read about agentic AI, you can for instance call an LLM multiple times with different security consideration everytime.

And that buys you what, exactly? Your point is 100% correct and why LLMs are no where near able to manage / build complete simple systems and surely not complex ones. Why? Context. LLMs, today, go off the rails fairly easily. As I've mentioned in prior comments I've been working a lot with different models and agentic coding systems. When a code base starts to approach 5k lines (building the entire codebase with an a…

To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too. It’s not more reliable than just taking a random person off the street of SF and giving them instructions on what to copy paste without any context. They might also change unrelated things or get sidetracked when they encounter friction. My point is that when you try to compensate by prompting repeatedly, you are just adding more chances for entropy to leak in — so I am agreeing with you.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#272

Post by Lord-Jobo »

eru wrote: post #153
theoldgreybeard wrote: post #146

The bigwigs at my company want to build out a document management suite. After talking to VP of technology about requirements I ask about security as well as what the regulatory requirements are and all I get is a blank stare. I used to think developers had to be supremely incompetent to end up with vulnerabilities like this. But now I understand it’s not the developers who are incompetent…

There's enough incompetence at all levels to go around.

Not only does the Peter principle generally show more incompetence the higher up a structure you move, but the outsized influence those positions have make for a very noticeably higher level of “things fucked up by incompetence“ coming from the C suite compared to the rest of the structure.

There’s definitely plenty of incompetence regardless. But I’ve never seen a company where the incompetence was more noteworthy in the cog positions than “leadership”.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#273

Post by dogman144 »

j45 wrote: post #145
dogman144 wrote: post #114

Earlier quoted context omitted.

Assuming a 101 security program past the quality bar, there are a number of reason why this can still happen at companies. Summarized as - security is about risk acceptance, not removal. There’s massive business pressure to risk accept AI. Risk acceptance usually means some sort of supplemental control that’s not the ideal but manages. There are very little of these with AI tools however - small vendors, they’re not…

You missed risk creation vs reward creation. And then folks can gasp and faint like goats and pretend they didn’t know. It reminds me of the time I met an IT manager who dint have an IT background. Outsourced hilarity ensued through sales people who were also non-technical.

What am I missing? Risk acceptance is what you’re referring to - risk creation and reward creation.

Sec lead might have a pretty darn clear idea of an out of whack creation of risk v reward. CEO disagrees. Risk accept and move on.

When you’re technical and eventually realize there’s a business to survive behind the tech skills, this is the stuff you learn how to do.

People “will know” as you say because it’s all documented and professionally escalated.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#274

Post by [deleted] »

Covenant0028 wrote: post #259
blitzar wrote: post #222

Earlier quoted context omitted.

humans used open s3 buckets stuffed with text files of usernames, passwords, addresses, credit card numbers etc long before vibe coding was a thing.

And those humans would be looking for a new job or face other consequences. An AI model can merrily do this with zero consequences because no meaningful consequences can be visited upon it. Just like if any human employee publicly sexually harassed his female CEO, he'd be out of a job and would find it very hard to find a new one. But Grok can do it and it's the CEO who ends up quitting.

[deleted]

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#275

Post by ethbr1 »

joshribakoff wrote: post #268
eric-burel wrote: post #251

Earlier quoted context omitted.

You may want to read about agentic AI, you can for instance call an LLM multiple times with different security consideration everytime.

You may want to try using it, anecdotes often differ from theories, especially when they are being sold to you for profit. It takes maybe a few days to see a pattern of ignoring simple instructions even when context is clean. Or one prompt fixes one issue and causes new issues, rinse and repeat. It requires human guidance in practice.

Strongman: LLMs aren't a tool, they're fuzzy automation.

And what keeps security problems from making it into prod in the real world?

Code review, testing, static and dynamic code scanning, and fuzzing.

Why aren't these things done?

Because there isn't enough people-time and expertise.

So in order for LLMs to improve security, they need to be able to improve our ability to do one of: code review, testing, static and dynamic code scanning, and fuzzing.

It seems very unlikely those forms of automation won't be improved in the near future by even the dumbest form of LLMs.

And if you offered CISOs a "pay to scan" service that actually worked cross-language and -platform (in contrast to most "only supported languages" scanners), they'd jump at it.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#276

Post by MangoToupe »

AbstractH24 wrote: post #263
MangoToupe wrote: post #217

Earlier quoted context omitted.

The salary you pay them, typically

Salaries make humans infallible?

No, but it makes them motivated to be thorough. There is no way to motivate a chatbot (to do better or to any end).

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#277

Post by j45 »

jimbokun wrote: post #67
j45 wrote: post #8

Earlier quoted context omitted.

It's a little hilarious. First, as an organization, do all this cybersecurity theatre, and then create an MCP/LLM wormhole that bypasses it all. All because non-technical folks wave their hands about AI and not understanding the most fundamental reality about LLM software being fundamentally so different than all the software before it that it becomes an unavoidable black hole. I'm also a little pleased I used two sp…

My first reaction to the announcement of MCP was that I must be missing something. Surely giving an LLM unlimited access to protected data is going to introduce security holes?

Agree.

It’s assuming and estimating it will behave like other software before it when it’s nothing like the software that came before it.

LLMs today won’t behave like the software we’re used to where 1+1 will equal 2 every time.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#278

Post by corry »

me_again wrote: post #126
corry wrote: post #73

"Companies often have a demo environment that is open" - huh? And... Margolis allowed this open demo environment to connect to their ENTIRE Box drive of millions of super sensitive documents? HUH???! Before you get to the terrible security practices of the vendor, you have to place a massive amount of blame on the IT team of Margolis for allowing the above. No amount of AI hype excuses that kind of professional misju…

I don't think we have enough information to conclude exactly what happened. But my read is the researcher was looking for demo.filevine.com and found margolis.filevine.com instead. The implication is that many other customers may have been vulnerable in the same way.

Ah, I see now that I read too quickly - the "open demo environment" was clearly referencing the idea that the vendor (Filevine) would have a live demo, NOT that each client wanted an open playground demo account that is linked to a subset of their data (which would be utterly insane).

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#279

Post by slop-cop »

jfindper wrote: post #93
anonymous908213 wrote: post #88

Earlier quoted context omitted.

LLMs will never get good enough that no one can tell the difference, because the technology is fundamentally incapable of it, nor will it ever completely disappear, because the technology has real use cases that can be run at a massive profit. Since LLMs are here to stay, what we actually need is for humans to get better at recognising LLM slop, and stop allowing our communication spaces to be rotted by slop articles…

Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot? The only spam I see in this chain is the flagged post by electric_muse. It's actually kind of ironic you bring up copy-paste spam bots. Because people fucking love to copy-paste "ai slop" on every comment and article that uses any punctuation rarer than a period.

[dead]

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#280

Post by windexh8er »

joshribakoff wrote: post #271
windexh8er wrote: post #262

Earlier quoted context omitted.

And that buys you what, exactly? Your point is 100% correct and why LLMs are no where near able to manage / build complete simple systems and surely not complex ones. Why? Context. LLMs, today, go off the rails fairly easily. As I've mentioned in prior comments I've been working a lot with different models and agentic coding systems. When a code base starts to approach 5k lines (building the entire codebase with an a…

To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too. It’s not more reliable than just taking a random person off the street of SF and giving them instructions on what to copy paste without any context. They might also change unrelated things or get sidetracked when they encounter friction. My point is that when you try to compensate by…

> To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too.

Spot on. If we look at, historically, "AI" (pre-LLM) the data sets were much more curated, cleaned and labeled. Look at CV, for example. Computer Vision is a prime example of how AI can easily go off the rails with respect to 1) garbage input data 2) biased input data. LLMs have these two as inputs in spades and in vast quantities. Has everyone forgotten about Google's classification of African American people in images [0]? Or, more hilariously - the fix [1]? Most people I talk to who are using LLMs think that the data being strung into these models has been fine tuned, hand picked, etc. In some cases for small models that were explicitly curated, sure. But in the context (no pun) of all the popular frontier models: no way in hell.

The one thing I'm really surprised nobody is talking about is the system prompt. Not in the manner of jailbreaking it or even extracting it. But I can't imagine that these system prompts aren't collecting mass tech debt at this point. I'm sure there's band aid after band aid of simple fixes to nudge the model in ever so different directions based on things that are, ultimately, out of the control of such a large culmination of random data. I can't wait to see how these long term issues crop and and duct taped for the quick fixes these tech behemoths are becoming known for.

[0] https://www.bbc.com/news/technology-33347866 [1] https://www.theguardian.com/technology/2018/jan/12/google-ra...

Post reply on HN