Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

alexschapiro.com

Post reply on HN
Print view View original

221–230 of 301 posts

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#221

Post by jll29 »

bongodongobob wrote: post #90
whstl wrote: post #77

Earlier quoted context omitted.

I've once had a whole sector of a fintech go down because one DevOps person ignored daily warning emails for three months that an API key was about to expire and needed reset. And of course nobody remembered the setup, and logging was only accessible by the same person, so figuring out also took weeks.

I'm currently on the other side of this trying to convince management that the maintenance that should have been done 3 years ago needs to get done. They need "justification".

Write a short memo that saying you are very concerned, and describe a range of things that may happen (from "not much" over medium to maximum scare - lawsuits, brand/customer trust destroyed etc.).

Email the memo to a decision maker with the important flag on and CC: another person as a witness.

If you have been saying it for a long time and nobody has taken any action, you may use the word "escalation" as part of the subject line.

If things hit the fan, it will also make sure that what drops from the fan falls on the right people, and not on you.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#222

Post by blitzar »

altmanaltman wrote: post #202
0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

humans used open s3 buckets stuffed with text files of usernames, passwords, addresses, credit card numbers etc long before vibe coding was a thing.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#223

Post by Tepix »

kylecazar wrote: post #6

If they have a billion dollar valuation, this fairly basic (and irresponsible) vulnerability could have cost them a billion dollars. If someone with malice had been in your shoes, in that industry, this probably wouldn't have been recoverable. Imagine a firm's entire client communications and discovery posted online. They should have given you some money.

Who says they didn't give him money?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#224

Post by blitzar »

theoldgreybeard wrote: post #163
eru wrote: post #153

Earlier quoted context omitted.

There's enough incompetence at all levels to go around.

Maybe I have just been lucky, but I have not had the displeasure of working with people either tha incompetent or willfully ignorant yet.

"Are We the Baddies?"

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#225

Post by objclxt »

gbacon wrote: post #218
0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

This is HN. We understood exactly what “exposed … confidential files” meant before reading your overly dramatic scenario. As overdone as it is, it’s not even realistic. A likely single mother is likely tiny potatoes in comparison to deep-pocketed legal firms or large corporations. The story is an example of the market self-correcting , but out comes this “building code” hobby horse anyway. All a software “building co…

> Building physical buildings is a much simpler, much less complex process with many fewer degrees of freedom than building software.

I don't...think this is true? Google has no problems shipping complex software projects, their London HQ is years behind schedule and vastly over budget.

Construction is really complex. These can be mega-projects with tens of thousands of people involved, where the consequences of failure are injury or even death. When software failure does have those consequences - things like aviation control software, or medical device firmware - engineers are held to a considerably higher standard.

> The private market is perfectly capable of performing this function

But it's totally not! There are so many examples in the construction space of private markets being wholly unable to perform quality control because there are financial incentives not to.

The reason building codes exist and are enforced by municipalities is because the private market is incapable of doing so.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#226

Post by xp84 »

Through that API, the frontend is handed a high-privilege API key for Box. Not only was that a huge blunder on the backend, but it reveals what passes for architecture these days. Should our application's backend speak to the super sensitive file store? No, we should hand over the keys to that to the React app, because it literally did not occur to them that there's anything that physically could be driven by the frontend that shouldn't be.

My apologies to the frontend engineers out there who know what they're doing.

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#227

Post by compootr »

altmanaltman wrote: post #202
0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

@grok all software engineers do is mindlessly turn specifications into code in one shot, right?!?

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#228

Post by Sharlin »

0xbadcafebee wrote: post #173

So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents . This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor. Imagine the potential impact. You're a single mother, fighting for custody of your kids. You…

Basically what happened in the Vastaamo case in Finland [1]. Except of course it wasn't individual phone calls – it was mass extortion of 30,000 people at once via email.

[1] https://en.wikipedia.org/wiki/Vastaamo_data_breach

Re: Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

#230

Post by dr_dshiv »

blitzar wrote: post #222
altmanaltman wrote: post #202

Earlier quoted context omitted.

but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...

humans used open s3 buckets stuffed with text files of usernames, passwords, addresses, credit card numbers etc long before vibe coding was a thing.

Prediction: Vibe coding systems will be better at security in 2 years than 90% of devs.
Post reply on HN