Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

1. The standards aren't clearly defined (i.e., you must specifically do this).

2. They are defined in terms of efforts rather than effects. It is like saying "every car sold must be made of steel" rather than "every car sold must be capable of withstanding an impact against a concrete wall at 60mph with X amount of deformation, etc." We want the rules to determine what level of threat is protected against, not just what motions the company went through. In the case in the article, it wasn't because hackers bypassed a bunch of safeguards; the company didn't protect against even basic threats.

3. It's not enough to have "liability". That puts the onus on individuals to sue the company for their specific damages. We need criminal penalties that are designed to punish companies (and the individuals who direct them) for the harm they do to society by the overall process of rushing ahead selling things instead of slowing down and being careful. We need large-scale enforcement so that companies actually stop doing these things because the cost of doing them becomes too enormous.

4. Our laws do not adequately take account of the differential power of those who cut corners, and the differential gains reaped. We frequently find small operators on the wrong end of painful lawsuits and onerous criminal penalties, while the biggest companies and wealthiest individuals use their position to avoid consequences. Laws need to explicitly take this into account, lowering the standard of proof for penalties against larger, wealthier, and more powerful companies and individuals, and also making those penalties exponentially higher.

"GenAI" is nothing new. "AI" is just software. It's not intelligent, or alive, or sentient, or aware. People can scifi sentimentalize it if they want.

It might simulate parts of things, hopefully more reliably.

It's however a different category of software which requires management that doesn't exist yet how it should.

Cybersecurity security theatre for me is using a web browser to secure and administer what was previously already done and creating new security holes from a web interface.

Then, bypassing it to allow unmanaged MCP access to internal data moats creating it's own universe of security vulnerabilities, full stop. In a secured and contained environment, using an MCP to access data to unlock insight is one thing.

It doesn't mean dont' use MCPs. It means the AI won't figure out what the user doesn't know about security around securing MCPs which is a far more massive vulnerability because users of AI have delegated their thinking to a statistics formula ("GenAI"), because it is so impressive on the surface, but no one is checking the work to make sure it stays that way. Managing quality however, is improving.

My comment is calling out effectively letting external paths have unadulterated access to your private and corporate data.

Data is the new moat. Not UI/UX/Software.

A wormhole that exposes your data makes it available for someone to put it into their data moat far too commonly, and also for it to be mis-interpretted.

One could only imagine that if OP wasn't the first to discover it, people could've generated tons of shared links for all kinds of folders, for instance, which would remain active even if they invalidated the API token.

Speaking of LLMs, did you notice the comment you were responding to was written by an account posting repetitive LLM-generated comments? :)