Why Law Firm Rankings Are Useless • Hacker News Board

Re: Why Law Firm Rankings Are Useless

#21

Post by savanaly » Thu, Sep 15, 2016, 7:33 PM UTC

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?

Re: Why Law Firm Rankings Are Useless

#22

Post by andrewfong » Thu, Sep 15, 2016, 7:37 PM UTC

wikiwawa wrote: post #11

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

A large part of statistics is devoted to making valid inference from noisy samples, largely by making sensible assumptions. Even in your example, it is common practice to assess a surgeon on outcomes of surgery, even though a lot of a surgeon's work does not lead to surgery. We aren't claiming to assess the wide practice of law, just litigation. We do Litigation Analytics.

What do you count as a "court judgment" and how are they evaluated? Are, for instance, a successful motion to dismiss and a jury decision for the defendant weighed equally? Or do you give higher weight to the motion to dismiss since that's usually a better outcome for the defendant than making it to trial?

And do you look at motions other than actual judgments? I.e. if Firm A wins a motion to suppress some piece of evidence and then there's a subsequent settlement, we should infer that Firm A "won" the case, even if there's no judgment to that effect.

Re: Why Law Firm Rankings Are Useless

#23

Post by rayiner » Thu, Sep 15, 2016, 7:41 PM UTC

andrewfong wrote: post #19

rayiner wrote: post #2

As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stac…

If I were interested in litigation funding, I wouldn't evaluate the ability of law firms to litigate but rather the ability of lawyers to accurate predict the outcome of cases (regardless of whether they're actually representing the client) -- e.g. associates at law firms frequently write memos to assess the probability of success before a case makes it way to trial. Seems like there's value in actually correlating m…

The article says they have a machine-learning algorithm that can "calculate the probability of a successful outcome" of a case. Then they calculate firms' ability to improve (or not) that outcome. But if I had such an algorithm I'd just use it directly to decide what cases to fund.

My understanding from talking to litigation funders is that lawyers are terrible at valuing cases. There is money to be made for anyone that can use technology to improve those predictions.

Re: Why Law Firm Rankings Are Useless

#24

Post by lordnacho » Thu, Sep 15, 2016, 7:42 PM UTC

robrenaud wrote: post #17

spitfire wrote: post #15

Why not use a modified Elo rating? That way you can not only easily figure out if you're going to win, but simulate the outcome of different scenarios?

What is a modified elo rating?

Elo rating is a system used to gauge chess players. You win/lose point depending on your points difference before a match. If the favourite loses, more points are lost to the underdog than if the underdog loses.

You can apply this to a wide variety of sports where most teams don't get to play most other teams, eg you can guess that Germany are better than India at soccer despite the rarity of the tie.

And of course you can modify the system to take account of various issues such as the retirement of better players, which tends to remove points from the system.

Re: Why Law Firm Rankings Are Useless

#25

Post by ramblenode » Thu, Sep 15, 2016, 7:46 PM UTC

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

> This post is a catastrophe.

This is a pretty ungenerous reading of the article. Your complaint seems to be mostly about the generalizability of the model (a legitimate complaint, as you've explained) rather than its statistical validity. It still provides some interesting insights about the performance of firms at trial. Notably, there isn't a correlation between favorable judgments in those cases which have gone to trial and traditional law firm rankings. That's surprising, even if it doesn't generalize to overall law firm "success".

Re: Why Law Firm Rankings Are Useless

#26

Post by jessaustin » Thu, Sep 15, 2016, 8:20 PM UTC

savanaly wrote: post #21

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?

Why should we expect a correlation? Why should we assume they even thought about the issue, if it isn't mentioned in their analysis?

Re: Why Law Firm Rankings Are Useless

#27

Post by [deleted] » Thu, Sep 15, 2016, 8:24 PM UTC

[deleted]

Re: Why Law Firm Rankings Are Useless

#28

Post by busterarm » Thu, Sep 15, 2016, 8:28 PM UTC

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

> This is particularly shocking given that litimetrics is a legal service provider.

Not so shocking. Law firm ranking services are their competition. They drew a conclusion from the data that supports their business.

Law firm rankings absolutely can provide a value. Both on the firm and client side of the transaction.

Imagine you're on the client side of a qui tam (Whistleblower) or Immigration case. Are you going to bring that case to a law firm that you don't have visibility into a) how the firm is trusted by their clients and b) how the law firm is regarded among other law firms? That would be crazy!

Heck, in a qui tam you can't even get quality analytics into how well a firm litigates a case because the best qui tam cases get picked up and tried by the Federal Government!

(Disclosure: I'm a dev at a law firm.)

Re: Why Law Firm Rankings Are Useless

#29

Post by wahern » Thu, Sep 15, 2016, 8:30 PM UTC

savanaly wrote: post #21

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?

Depends on what is meant by bullshit. Yes, if it meant that the study was absolutely wrong. No, if it meant that the study failed to provide any evidence whatsoever that there exists a correlation.

In science guess who has the burden?

In a legal context the burden of proof is broken down into the burden of production and the burden of persuasion. If you fail to meet the burden of production it's fair to say a claim is bullsh*t. It necessarily follows that if you haven't met the burden of production you cannot have met the burden of persuasion as a logical matter.

Remember, any claim must stand on its own terms. It doesn't matter what is the reality or truth of the matter. We can never know the absolute truth of something. We can only attain a slightly firmer grip on reality when people propose persuasive arguments. Maybe you can mine a failed argument for useful bits, but a good rule of thumb is that it's not worth the time if the argument cannot even meet a minimal burden. If it wasn't worth the claimant's time why would it be worth yours? That's strong circumstantial evidence to move on.

Re: Why Law Firm Rankings Are Useless

#30

Post by downandout » Thu, Sep 15, 2016, 8:31 PM UTC

libertymcateer wrote: post #4

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-…

While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. This is the same metric that the other side will primarily look at when considering the terms of any settlement.

Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.