Why Law Firm Rankings Are Useless
blog.litimetrics.com
Why Law Firm Rankings Are Useless
1–10 of 64 posts
Re: Why Law Firm Rankings Are Useless
#2As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stack the deck against plaintiffs. Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm.
According to the article, their analysis does account for that. They allegedly "calculate the probability of a successful outcome for the applicant" using machine learning. But there is no description of how they do that. If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.
Re: Why Law Firm Rankings Are Useless
#3>> "the job of a lawyer is to persuade, which is part legal reasoning and part hard work. Since the ability to persuade directly affects litigation outcomes, participants are willing to pay highly regarded barristers upwards of $28,000 a day, and some law firm partners an hourly rate of over $1,000. In this post, we analyse data from 22,000 cases to measure the differential impact law firms and barristers have on litigation outcomes."
No. That's what TV lawyers are paid for. That's the equivalent of learning about doctors from watching MASH. Most doctors are not surgeons. Most lawyers are not litigators. Many great and valuable lawyers will never win any case. Imho the best lawyer is the one who's client never faces the uncertainty and protracted negativity that is modern litigation, just as a great and valuable doctor is one who's patients avoid the knife. (Apples and oranges but the point stands.)
Lawyering is, in my world, mostly about listening to clients and understanding how their realities fit into external standards such as laws and industry norms. It's about hedging, documentation and interpretations. And sometimes it's about holding the client's feet to a fire. Litigators only appear once all of that has failed. Litigators may have the highest hourly rates, but do not have the highest salaries overall. Those are reserved for corporate counsels and board members.
Re: Why Law Firm Rankings Are Useless
#4> By applying machine learning algorithms to data mined from court judgments...
And, with that sentence, this post becomes bullshit.
The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned.
Even at that, only on the order of a single digit-percentage of disputes ever reaches a verdict.
This is like judging a hospital-system based on its success-rate for transplant surgery. It's just deeply ignorant of what constitutes the vast, vast majority of the practice of law - even in disputes.
This article has no value, whatsoever, aside from a witty aside at a cocktail party about how deeply misunderstood the legal profession is. This is particularly shocking given that litimetrics is a legal service provider.
> Although A has not faced B, if both have faced C, then we have some information about the hypothetical A and B matchup. This is the same reasoning that allows one to compare tennis players across different eras.
The implication is that the lawyer efficacy, based on judgments, is subject to the transitive property? Sweet mercy that is ridiculous.
This post is a catastrophe.
Re: Why Law Firm Rankings Are Useless
#5As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stac…
> Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm.
Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm.
Re: Why Law Firm Rankings Are Useless
#6> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost.
That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to firms.
Re: Why Law Firm Rankings Are Useless
#7"To sidestep the randomness of observed outcomes, we focus on modeling the probability of a successful outcome for a given case. Secondly, to account for the context or difficulty of the case, we measure the change in the probability of a successful litigation outcome, from substituting a legal service provider in. The difficulty of the case is not important, but rather the impact a legal service provider has on that original probability of success. From now on, we will refer to this measure as the impact on expected performance. This substitution exercise is known as a counterfactual, as we apply the substitutions to historical data."
-----------------
how exactly did they model the probability of successful outcomes for given cases?
because those depend on highly detailed fact patterns, many of which are largely qualitative in nature
if they were actually able to put all those in a data base that you could then model
that'd be a more interesting accomplishment to read about
Re: Why Law Firm Rankings Are Useless
#8> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost. That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to f…
Hell, if the lawyers have done a good job, both sides walk away from a settlement thinking that they won.
Re: Why Law Firm Rankings Are Useless
#9As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stac…
> Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm. Not a problem for the analysis of law firms . Sierra Club is a less effective law firm than the big New York firm.
> Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm.
Why is this not a problem? The parent comment was saying that the Sierra Club takes harder cases and is often the plaintiff, not the defendant, so they may be more effective as a law firm but have worse numbers because the win/loss numbers don't mean the same things in different contexts.
Re: Why Law Firm Rankings Are Useless
#10>> "the job of a lawyer is to persuade, which is part legal reasoning and part hard work. Since the ability to persuade directly affects litigation outcomes, participants are willing to pay highly regarded barristers upwards of $28,000 a day, and some law firm partners an hourly rate of over $1,000. In this post, we analyse data from 22,000 cases to measure the differential impact law firms and barristers have on lit…
Depends on the subject matter and the willingness of a plaintiff and defendant to negotiate a fair settlement.
In some areas, big money comes with big litigation.