I'm going to need some legal help for my startup. But I can't pay much. So I figured I will ask AI all relevant questions, as well as forms filled etc. Perhaps even create a patent-application for me. THEN I find a human lawyer and give AI's answers to them and say "Can you find any errors in this? Can you improve it?" . That way I think my legal bills should be smaller because the AI has already done most of the wor…
i use codex to do initial research and draft texts (in typst). i use files-output skill so that all research contexts are rendered into files md files. i do second phase on codex, by asking to download all pdfs and extract all text of laws it references. can repeat fully local research step. after i ask gemini to find issues and criticize. UPDATE: there many legal skills on github to try, not used so any yet
AI outperforms law professors in Stanford Law study
111–120 of 384 posts
Re: AI outperforms law professors in Stanford Law study
#112I'm going to need some legal help for my startup. But I can't pay much. So I figured I will ask AI all relevant questions, as well as forms filled etc. Perhaps even create a patent-application for me. THEN I find a human lawyer and give AI's answers to them and say "Can you find any errors in this? Can you improve it?" . That way I think my legal bills should be smaller because the AI has already done most of the wor…
While you might find a lawyer who bills fewer hours, think about what would happen if you got a second legal opinion from another human lawyer. The second lawyer would still need to do essentially the same work. In fact, their ethics would likely require them to independently review the facts, documents, and legal issues before giving you advice.
So even if AI gives you a draft, the lawyer is not simply checking grammar or spotting obvious mistakes. They still have to verify the analysis, look for missing issues, and decide whether the work is legally sound. Could even be more expensive
Re: AI outperforms law professors in Stanford Law study
#113As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.
Re: AI outperforms law professors in Stanford Law study
#114What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated…
Same is probably true of humans. In a conversation, we often respond from instinct, then work backwards to a rationalization only when asked. For more considered thoughts, if we’re lucky, we can remember our “reasoning traces” but that’s as deep as our introspection goes. Unless we’re neuroscientists, we don’t even know how many neurons we have, let alone have any understanding of how they generate our thoughts. Moti…
Re: AI outperforms law professors in Stanford Law study
#115Earlier quoted context omitted.
IME so far (as both a lawyer and a software engineer), LLM error rates when drafting code and legal documents are reasonably comparable, but it's more problematic in the legal context because legal documents do not benefit from many of the structural safeguards available for code. For legal documents, there are no automated tests, no static typing, no test environments, no logging/observability instrumentation, no sa…
Well this is largely the fault of law itself. especially english style law. A legal, parseable code, in which not every single tiny municipality (some less than 1 square mile) has their own set of rules and laws, not all published or available - but which citizens are expected to abide by of course - how could we expect AI to do well and not some typical TV southern lawyer who knows the judge?
For murder that's not such a huge deal because the statutes are typically easy to track down and don't really differ all that much substantively, but once you get really into the weeds on something like commercial contracts it can be a huge pain to do cross-jurisdictional research.
And that's just a tiny, super obvious example of how impenetrable statutory law is, which isn't even the really pernicious problem. Case law is infinitely worse. It makes me absolutely furious how difficult legal research still is. The Westlaw/LexisNexis duopoly is a moral crime and wildly destructive to the quality of government in this country. Every single written court opinion should be publicly available for free on the internet in an easily searched format. It would cost practically nothing to achieve. We're talking about less text than Wikipedia hosts. Yet still many states make it almost impossible to access case law. Even though these cases are law. Binding law that we are supposed to follow, yet we cannot even easily access. It's insane, and largely perpetuated by the complacency of lawyers who can charge others for what should be free, the lobbying of the duopoly, and the incompetence of politicians.
If all of the laws were consistently available and stored in reasonable, consistent citation formats (I would settle for hyperlinking as a replacement for the rat's nest of wildly varying jurisdiction-specific citation systems), it would even be possible to introduce a form of unit testing for legal drafting that would allow us to automatically verify if the LLM hallucinated a citation.
It also doesn't help that we (for what were at the time very good reasons) moved away from the system of legal writs that used to provide fairly standardized, almost "cut and paste" templates for legal filings. So now every legal document (filings, memos, contracts, court opinions, statutes) is drafted like a bespoke, artisanal creation with few strict structural or stylistic conventions. That makes automated interpretation much harder than it needs to be.
Re: AI outperforms law professors in Stanford Law study
#116What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated…
There are however LLM context building techniques that anchor completions in data structures that persist the structure of claims that support the conclusion contained in a completion. Lots of different patterns exist —organizing logic in language is a rich domain— but the one I’ve liked the most is something called a Claim Dependency Graph that models the relationships between atomic claims as graph edges.
There’s a whole suite of operations you can perform on these structures, and “reconstruct how you came to this conclusion” is absolutely one of them.
Re: AI outperforms law professors in Stanford Law study
#117Earlier quoted context omitted.
Completely unevidenced conspiracy theories can only harm the discussion. The only possible benefit is to disconfirm conspiracy theories and discourage paranoid thinking. The odds that Standford as an institution are astroturfing on HN round down to 0. What they're almost certainly observing is that these critical comments are being flagged as inappropriate. People make inappropriate comments that happen to contain cr…
may I ask why you effectively said 'conversation over due to harm reasons' instead of asking for evidence to support the conclusion that you believe is not possible? I don't see why it is inherently harmful to discuss the seemingly impossible. I also don't see why it's relevant to bring up your n=1 sample (although it is as relevant as my n=1 sample, which has plenty of astroturfing witnessing [unspecific to Stanford…
Re: AI outperforms law professors in Stanford Law study
#118As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.
Absolutely not harmless if you're the executor of an estate forced to deal with a screwed up AI will. I just handler my dad's estate this spring. It's a frustrating and confusing process even with the simplest of estates.
Re: AI outperforms law professors in Stanford Law study
#119Earlier quoted context omitted.
If you want human connection the legal system is not where you are going to find it, period. I don't think there will be any such market for "non ai" law. If I'm involved with the legal system I just want out as quick as possible as cheap as possible.
Bad legal advice will keep you dealing with the legal system for much longer and at much greater cost. Something being cheap and quick upfront doesn't mean it will be cheap and quick by the end of the process.
Even the good ones will not step above and beyond what they are paid to do
but an AI ? it will and can go above and beyond
Re: AI outperforms law professors in Stanford Law study
#120In the framing of using LLMs as legal tutors, with the implication of lowering the cost of legal training, this seems like a socially-positive outcome. Furthermore, it feels kind of intuitive to me that any contemporary system operating with an LLM and access to legal reference material will be prepared to answer _student-originated questions_ comprehensively and with breadcrumbs or direct references to educational/source materials, as seems to have been found in the study.
The authors explicitly and intentionally emphasize that many legal questions require contextualization, as opposed to some discrete calculated answer. The result of the study implies that the LLM-based systems were capable of using what many of us here understand to be the "stochastic best-fit algorithmic generation" of a contemporary language model to adequately contextualize a student's question, providing insight into the trade-offs or complications implicit in the question, while then, critically, _meeting the professional standards of legal educators in explaining that complexity to a student_.
Realistically, I would hope this provides some confidence to readers of HN that they can actually ask a legal question to an LLM and expect the response will explain the complexity of the law in relation to the question. This is great news, and is likely the minimal pre-work any of us should do before actually consulting a lawyer, if time permits.
On the other hand, I do _not_ think that this study provides any indication that an LLM is prepared to actually provide direct legal counsel. Possibly in the same way that a legal textbook does not replace legal counsel, or perhaps more accurately, the same way that stumbling upon a legal case study for approximately the same situation you're in doesn't guarantee you'll have the same result.