I worked on applying machine learning algorithms to data mined from court judgments during my PhD. You want to know what we learned? That it's better to be white and rich than black and poor. Point. Everything else was bullshit.
Why Law Firm Rankings Are Useless
31–40 of 64 posts
Re: Why Law Firm Rankings Are Useless
#32Also, being a Lawyer, if someone comes to me, it's because of me, not because of the firm I am working in. And trying to rank lawyers? Why not. But what's the point. I wouldn't accept being ranked anyway.
Re: Why Law Firm Rankings Are Useless
#33> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. This is the same metric that the other side will primarily look at when considering the terms of any settlement. Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.
Having spent the last 12 years of my life wrapped up in a civil trial related to estate law and learned a ton along the way about how lawyers have failed me, I would much rather have known how often a firm's cases go to trial and if not, which way the case went.
It will take _years_ before your case actually goes to trial and you will meet plenty of lawyers along the way if you aren't careful who will happily fuck off with your money. And you'll have to sue them separately to deal with that, if you even have the energy to spare.
Good luck having a full-time job through that mess.
Re: Why Law Firm Rankings Are Useless
#34> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
A large part of statistics is devoted to making valid inference from noisy samples, largely by making sensible assumptions. Even in your example, it is common practice to assess a surgeon on outcomes of surgery, even though a lot of a surgeon's work does not lead to surgery. We aren't claiming to assess the wide practice of law, just litigation. We do Litigation Analytics.
So you sue me and get a judgment for $100,000.
If I felt that I owed you nothing, surely this counts as a "loss" for me at trial.
If, however, you had originally sued for $50M and I felt I might owe you some money but no more than $1M so I decided to fight it, the $100k judgment would be considered an enormous "win" for me.
What data does your analysis use to distinguish between the two scenarios given that the verdicts are identical?
Re: Why Law Firm Rankings Are Useless
#35> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?
Yes, it does, and yes, that is correct.
Lawyers can drop clients with cases that they think are losers and refuse to take them to trial. And, very, very often, before or directly after closing arguments, if the writing is on the wall, a case settles.
Additionally, what sorts of cases are we talking about? Insurance defense? Toxic torts? Data breach?
The 'winnability' of these cases varies highly. There are many, many law firms who's entire business model is built entirely only on taking no-brainer winning cases. Think injury lawyers. They may - perversely - have very low winning percentages, because 90-95% of their cases settle, and only the real squeakers get to trial.
Let's look at the inverse. Do you really need Quinn Emmanuel or Gibson Dunn if you have a slam-dunk case? Or do you need the best litigators around when your case is a total coin-toss? And, in that event, is it an example of bad lawyering if your crack-team loses because they are pushing the bounds of appellate advocacy?
The idea of judging 'law firms' without further taxonomic distinction, generally, by trial disposition is just - it is frankly idiotic.
Re: Why Law Firm Rankings Are Useless
#36> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. This is the same metric that the other side will primarily look at when considering the terms of any settlement. Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.
> While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm.
Read above to my last reply.
This is really a poor way to think about law firms.
Is the firm a toxic tort firm? Is it an appellate advocacy firm? Is it maritime law? Is it an intellectual property dispute? A business dispute?
Win percentages are totally meaningless without more information, including client-positive settlement rates.
This metric is totally useless.
Re: Why Law Firm Rankings Are Useless
#37Earlier quoted context omitted.
I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost. That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to f…
Hell, if the lawyers have done a good job, both sides walk away from a settlement thinking that they won.
Or that neither won.
Re: Why Law Firm Rankings Are Useless
#38Earlier quoted context omitted.
While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. This is the same metric that the other side will primarily look at when considering the terms of any settlement. Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.
> While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. Read above to my last reply. This is really a poor way to think about law firms. Is the firm a toxic tort firm? Is it an appellate advocacy firm? Is it maritime law? Is it an intellectual property dispute? A business dispute? Win percentages are totally meaningless without more information, includ…
>This metric is totally useless.
You honestly believe that an opponent won't take a 90%+ trial win rate into consideration when considering a settlement? All you hear from lawyers pretrial is "our chances at trial are X" and then based on whether X is high or low, they'll make a decision to go to trial or settle.
Re: Why Law Firm Rankings Are Useless
#39As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stac…
> If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.
Legalist is a recent YC startup aiming to do exactly that:
http://www.newyorker.com/business/currency/what-litigation-f...
Re: Why Law Firm Rankings Are Useless
#40> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…
"The overwhelming majority of disputes do not reach litigation. ..."
I'm shopping for a personal injury lawyer. How to pick? The two referrals I got didn't pan out. I don't know any lawyers. Aha, search the court documents to see who the winners are!
I (finally) consulted my paralegal friend. He made the same points you have.
Additionally, he said the courts all use different IT systems. One could use an expensive service to do the search, but it's hardly worth it for laypersons.
Too bad.
I did start to wonder if one could data mine the filings, just to see who's getting the most cases, and maybe infer something actionable from that.