Live data from Hacker News

Answering Legal Questions with LLMs

hugodutka.com

41–50 of 160 posts

Re: Answering Legal Questions with LLMs

#41
We’ve learned that the combination of high latency, faulty reasoning, and limited document scope kills usage. No lawyer wants to expend effort to ask a detailed question, wait 10 minutes for an answer, wade through a 2-pages-long response, and find that the AI made an error.

Nor does any lawyer want to have that same experience with a junior associate (except insert “two hours” for “10 minutes”), yet here we are.

Re: Answering Legal Questions with LLMs

#42

The next step after this is more complicated laws because lawyers can now use LLMs, and thus laws even more opaque to ordinary folk who will have to use LLMs to understand anything. It's an even more fragile system that will undoubtedly be in favour of those who can wield the most powerful LLM, or in other words, the rich and the corporations. This is another example of technology making things temporarily easier, un…

You can solve it be assigning a complexity score to a law. If the law increases complexity you need a supermajority to pass it, otherwise simple majority is ok.

Re: Answering Legal Questions with LLMs

#43
I wonder if this "AI will replace your job" is like "AI will drive your car" in that where once something can solve 95% of the problem the general public assumes the last 5% will come very quickly.

Rodney Brooks used to point out that self-driving was perceived by the public as happening very quickly, when he could show early examples in Germany from the 1950s. We all know this kind of AI has been in development a long time and it keeps improving. But people may be overestimating what it can do in the next five years -- like they did with cars.

Re: Answering Legal Questions with LLMs

#44
post #16

Earlier quoted context omitted.

Is the resulting (C) code maintainable, unit testable, do you understand it? If your answer is "I'll just ask gemini to explain it", I will laugh sarcastically and then sob for the poor people around the hardware you program for

I haven't had an issue (at least more than what is expected). I am also an EE, not an SWE. I use it for internal test systems and it has saved me tons of time that I would have had to spend combing the reference manual. As I am sure you know, embedded code often has terrible portability and requires lots of "check the 500 page datasheet" to get stuff working properly.

Interesting, thank you!

Re: Answering Legal Questions with LLMs

#46
EU law is case driven, and besides the text of cases you also need to know the books interpreting them, general legal principles that might be applicable and hermeneutic traditions.

They are clearly a long way from a tool that can compete with a human lawyer.

Re: Answering Legal Questions with LLMs

#47
I would say that it’s getting better at answering those questions. I have a list of difficult legal research questions that I worked on at work and gemeni pro and claude opus are definitely way better than 3 and 3.5 and 4.

I believe it will eventually get there and give good advice.

Re: Answering Legal Questions with LLMs

#48

I would say that it’s getting better at answering those questions. I have a list of difficult legal research questions that I worked on at work and gemeni pro and claude opus are definitely way better than 3 and 3.5 and 4. I believe it will eventually get there and give good advice.

What is the situation regarding LLM access to the major repositories of case law and legislation at places like Westlaw/LexisNexis? Those are really basic information sources for lawyers around the world and access is tightly controlled (and expsensive!), but its enormously common for lawyers and law students to need subscriptions to those services.

I'm just curious because I can't imagine either Westlaw or LexisNexis giving being controller of access to this information up without a fight, and a legal LLM that isn't trained on these sources would be... questionable - they are key sources.

The legislation text can probably be obtained through other channels for free, but the case law records those companies have are just as critical especially in Common Law legal systems - just having the text of the legislation isn't enough for most Common Law systems to gain an understanding of the law.

EDIT: Looks like westlaw are trying their own solution, which is what I would have guessed: https://legal.thomsonreuters.com/en/products/westlaw-edge

Re: Answering Legal Questions with LLMs

#49

Naively I wonder if the tendency towards "plausible bullsh*t" could be problem here? Making very convincing legal arguments that rest of precedents that don't exist etc.

wonder no more [1]

In a cringe-inducing court hearing, a lawyer who relied on A.I. to craft a motion full of made-up case law said he “did not comprehend” that the chat bot could lead him astray.

[1] https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-s...

Re: Answering Legal Questions with LLMs

#50
> We preprocess the regulation so that when a call contains a reference to “Annex III,” we know which pages to put into the “junior lawyer’s” prompt. This is the LLM-based RAG I mentioned in the introduction.

Is this RAG or just an iteration on more creative prompt engineering?

Post reply on HN