Answering Legal Questions with LLMs
hugodutka.com
Answering Legal Questions with LLMs
1–10 of 160 posts
Re: Answering Legal Questions with LLMs
#2No thank you, let's not.
Re: Answering Legal Questions with LLMs
#3Don't sections of regulations reference each other, and reference other regulations? This article says they only insert snippets of the section they believe to be directly relevant to the legal question. It seems to me that this automatically puts the bot in a position where it lacks all the information it needs to construct an informed answer. Or are the laws in some regions drafted in a "stand-alone" way where each section is fully independent by restating everything?
This feels like they've built an ai that justifies itself with shallow quotes instead of a deep understanding of what the law means in context.
Re: Answering Legal Questions with LLMs
#4A tool like this should live in service to the legal profession. Like Copilot, without a human verifying, improving, and maintaining the work, it is risky (possibly negligent) to provide this service to end users.
Re: Answering Legal Questions with LLMs
#5Many laws, especially GDPR, can only be interpreted in conjunction with a lot of guidelines (WP29 for example), interpretations by the local Data Protection Authority, decisions by local and European courts, etc.
Given all of this information, I think the bot will be able to formulate and answer. However, the bot first needs to know what information is needed.
If a lawyer has to feed the bot certain specific parts of all of these documents, they might as well write the answer down themselves.
Re: Answering Legal Questions with LLMs
#6Don't sections of regulations reference each other, and reference other regulations? This article says they only insert snippets of the section they believe to be directly relevant to the legal question. It seems to me that this automatically puts the bot in a position where it lacks all the information it needs to construct an informed answer. Or are the laws in some regions drafted in a "stand-alone" way where each…
You're right that sections reference each other, and sometimes reference other regulations. By creating the "plan for the junior lawyer", the LLM can reference multiple related sections at the same time. In the second step of the example plan in the post there's a reference to "Articles 8-15", meaning 7 articles that should be analyzed together.
The system is indeed limited in the way that it cannot reference other regulations. We've heard it's a problem from users too.
Re: Answering Legal Questions with LLMs
#7A tool like this should live in service to the legal profession. Like Copilot, without a human verifying, improving, and maintaining the work, it is risky (possibly negligent) to provide this service to end users.
My number one request is still: "please rewrite this legal answer in simple language with short sentences." For this, it is amazing (as long as I proofread the result). For actual answers, eh...
Re: Answering Legal Questions with LLMs
#8A tool like this should live in service to the legal profession. Like Copilot, without a human verifying, improving, and maintaining the work, it is risky (possibly negligent) to provide this service to end users.
My number one request is still: "please rewrite this legal answer in simple language with short sentences." For this, it is amazing (as long as I proofread the result). For actual answers, eh...
I assume you're in the legal profession? Do you find, as you proofread, that you want to insert caveats and qualifiers into the "simple" language and end up with something like the original legalese?
Legal language is what it is in large part because simple short sentences are too imprecise to express the detail needed.
Re: Answering Legal Questions with LLMs
#9Unintuitive LLM only rag?
Re: Answering Legal Questions with LLMs
#10GPT-4 also cannot solve full programming problems, and frequently makes large errors even with a small focused context, as in Github Copilot Chat.
However, it is still extremely useful and productivity enhancing. When combined with the right workflow and UI. Programming is large enough of an industry, that has Microsoft building it out in VScode. I don't think the legal industry has a similar tool.
Also, I think programmers are far more sensitive to radical changes. They see the constant leaps in performance, and are jumping in to use the AI tools, because they know what could be coming next with GPT-5. Lawyers are generally risk averse, not prone to hype, so far less eager customers for these new tools.