This was an excellent article describing how they broke down a complex task that an LLM was bad at, into a series of steps that the LLM could excel at. I think that this approach is probably broadly applicable across law (and perhaps medicine).
Answering Legal Questions with LLMs
21–30 of 160 posts
Re: Answering Legal Questions with LLMs
#22The next step after this is more complicated laws because lawyers can now use LLMs, and thus laws even more opaque to ordinary folk who will have to use LLMs to understand anything. It's an even more fragile system that will undoubtedly be in favour of those who can wield the most powerful LLM, or in other words, the rich and the corporations. This is another example of technology making things temporarily easier, un…
Lmao dont make up laws like that please. If anything my guess is that LLMs will make laws simpler and without loopholes and rich people wont be able to hire lawyers to have a competitive advantage in exploiting legal loopholes
Isn't it true though, at least in terms of the amount of information we have to wade through these days? Haven't hard drives gotten larger, and why? Because technology makes it possible. It's funny that you are laughing, but it would be even better if you made a serious argument against me.
To be honest, I am posing a serious possibility: are we really sure that AI will cause a democratization of knowledge? I mean, our society is valuable and keeps us alive, so shouldn't we be at least asking the question?
It seems like even questioning technology around here is taboo. What's wrong with discussing it openly? I think it's rather naive to believe that technology will make life simpler for the average person. I've lived long enough to know that many inventions, such as the internet and smartphone, have NOT made life easier at all for many, although they bring superficial conveniences.
Re: Answering Legal Questions with LLMs
#23Is there perhaps a training data problem?
Even if the LLM were trained on the entire legal case law corpus, legal cases are not structured in a way that an LLM can follow. They reference distant case law as a reason for a ruling, they likely don't explain specifically how presented evidence meets various bars. There are then cross-cutting legal concepts like spoliation that obviate the need for evidence or deductive reasoning in areas.
I think a similar issue likely exists in highly technical areas like protocol standards. I don't think that an LLM, given 15,000 pages of 5G specifications, can tell you why a particular part of the spec says something, or given an observed misbehavior of a system, which parts of the spec are likely violated.
Re: Answering Legal Questions with LLMs
#24The next step after this is more complicated laws because lawyers can now use LLMs, and thus laws even more opaque to ordinary folk who will have to use LLMs to understand anything. It's an even more fragile system that will undoubtedly be in favour of those who can wield the most powerful LLM, or in other words, the rich and the corporations. This is another example of technology making things temporarily easier, un…
In the US, the vast majority of legislators are lawyers. Lawyers have their own “unions” (eg the American Bar Association”).
I can definitely see this kind of protectionism occurring.
OTOH, I also see potential for a proliferation of law firms offering online services that are LLM-driven for specific scenarios, or tech firms (LegalZoom etc) offering similar services, and hiring a lawyer on staff to ensure that they can’t be sued for providing unlicensed legal advice.
In other words it might compete with lawyers at the low end, but big law could co-opt it to take advantage of efficiency increases over hiring interns and junior lawyers.
Re: Answering Legal Questions with LLMs
#25I saw a RAG demo from a startup that allows you to upload patient's medical docs, then the doctor can ask it questions like:
> what's the patient's bp?
even questions about drugs, histories, interactions, etc. The AI keeps in mind the patient's age and condition in its responses, when recommending things, etc. It reminded me of a time I was at the ER for a rib injury and could see my doctor Wikipedia'ing stuff - couldn't believe they used so much Wikipedia to get their answers. This at least seems like an upgrade from that.
I can imagine the same thing with laws. Preload a city's, county's etc. entire set of laws and for a sentencing, upload a defendant's criminal history report, plea, and other info then the DA/judge/whoever can ask questions to the AI legal advisor just like the doctor does with patient docs.
I mention this because RAG is perfect for these kinds of use cases, where you really can't afford the hallucination - where you need its information to be based on specific cases - specific information.
I used to think AI would replace doctors before nurses, and lawyers before court clerks - now I think it's the other way around. The doctor, the lawyer - like the software engineer - will simply be more powerful than ever and have lower overhead. The lower-down jobs will get eaten, never the knowledge work.
Re: Answering Legal Questions with LLMs
#26I saw a RAG demo from a startup that allows you to upload patient's medical docs, then the doctor can ask it questions like: > what's the patient's bp? even questions about drugs, histories, interactions, etc. The AI keeps in mind the patient's age and condition in its responses, when recommending things, etc. It reminded me of a time I was at the ER for a rib injury and could see my doctor Wikipedia'ing stuff - coul…
I've 100% found AI to be super helpful in learning a new programming language or refreshing on one I haven't used in a while. Hey how do I this thing in Gleam? What's Gleams equivalent of y? I turn it first instead of forums/stackoverflow/google now and would say I only need to turn to other sources less than maybe 5% of the time.
Re: Answering Legal Questions with LLMs
#27Earlier quoted context omitted.
Lmao dont make up laws like that please. If anything my guess is that LLMs will make laws simpler and without loopholes and rich people wont be able to hire lawyers to have a competitive advantage in exploiting legal loopholes
Isn't it true though, at least in terms of the amount of information we have to wade through these days? Haven't hard drives gotten larger, and why? Because technology makes it possible. It's funny that you are laughing, but it would be even better if you made a serious argument against me. To be honest, I am posing a serious possibility: are we really sure that AI will cause a democratization of knowledge? I mean, o…
Look i think its true that technology makes life worse in many ways but making the legal system complex is not one of those things in my opinion.
There is nothing wrong with your position its just that you are trying to make weak generalizations driven primarily by your emotions and anecdotal experience and not data
Re: Answering Legal Questions with LLMs
#28Earlier quoted context omitted.
I'm surprised Gemini 1.5 isn't getting more attention. Despite being marginally worse than the leaders, its still solid and you can dump 975,000 (!) tokens into it and still have ~75,000 to play with. I've been using it lately for microcontoller coding, and I can just dump the whole 500 page MCU reference manual into it before starting, and it gives tailored code for the specific MCU I am using. Total game changer.
Is the resulting (C) code maintainable, unit testable, do you understand it? If your answer is "I'll just ask gemini to explain it", I will laugh sarcastically and then sob for the poor people around the hardware you program for
I haven't had an issue (at least more than what is expected). I am also an EE, not an SWE. I use it for internal test systems and it has saved me tons of time that I would have had to spend combing the reference manual.
As I am sure you know, embedded code often has terrible portability and requires lots of "check the 500 page datasheet" to get stuff working properly.
Re: Answering Legal Questions with LLMs
#29Earlier quoted context omitted.
Isn't it true though, at least in terms of the amount of information we have to wade through these days? Haven't hard drives gotten larger, and why? Because technology makes it possible. It's funny that you are laughing, but it would be even better if you made a serious argument against me. To be honest, I am posing a serious possibility: are we really sure that AI will cause a democratization of knowledge? I mean, o…
Look i think its true that technology makes life worse in many ways but making the legal system complex is not one of those things in my opinion. There is nothing wrong with your position its just that you are trying to make weak generalizations driven primarily by your emotions and anecdotal experience and not data
Well, I look forward to an easy refutation then.
Re: Answering Legal Questions with LLMs
#30I saw a RAG demo from a startup that allows you to upload patient's medical docs, then the doctor can ask it questions like: > what's the patient's bp? even questions about drugs, histories, interactions, etc. The AI keeps in mind the patient's age and condition in its responses, when recommending things, etc. It reminded me of a time I was at the ER for a rib injury and could see my doctor Wikipedia'ing stuff - coul…
I've 100% found AI to be super helpful in learning a new programming language or refreshing on one I haven't used in a while. Hey how do I this thing in Gleam? What's Gleams equivalent of y? I turn it first instead of forums/stackoverflow/google now and would say I only need to turn to other sources less than maybe 5% of the time.
I think that is right. The sweat spot is twofold: 1) A replacement for general search on a topic where you have limited familiarity that can give you an answer for a concise question, or a starting point for more investigation or 2) For power-user use cases, where there already exists subject matter expertise, elaboration or extrapolation from a clear starting point to a clear end state, such as translation or contextualized exposition.
The problem comes with thinking you can bridge both of those use cases - vague task descriptions to final output. The work described in the article of getting an LLM itself to break down a task seems to work sometime but struggles in many scenarios. Products that can define their domain narrowly enough, and embed enough domain knowledge into the system, and can ask the feedback at the right points, and going to be successful and more generalized systems will either need to act more like tools rather than complete solutions.