Statistical ranking of defense lawyers? Maybe, but not this way.

It’s an intriguing notion: that one can objectively assess the relative effectiveness of a given lawyer. With hard data, and sound analysis. In the real world, it’s nigh impossible to tell how good a lawyer really is. You can look on Avvo and see what people here and there may have subjectively thought about him, but that doesn’t tell you whether any other lawyer would have done as well (or been just as dissatisfying). You can ask around and get a sense of what other lawyers generally think of him, but that’s just as subjective. There’s really nothing out there to tell you for sure whether that lawyer gets better-than-average results or not.

So Wake Forest professors Ronald Wright and Ralph Peeples — to their great credit — tried to see if it could be done. In their recent paper, “Criminal Defense Lawyer Moneyball: A Demonstration Project,” they conclude that it can be done. They may even be right about that. But not from the data they gathered, sadly.

[Warning: The internet's gonna try to do it anyway. Granted, the authors are concerned more with providing a useful tool for managers of institutional organizations providing legal aid, than with helping potential clients assess a lawyer's abilities. But you can bet that Avvo or someone like them would love to develop an algorithm that ranks lawyers in a way that potential clients are willing to pay for. It may or may not be all that objectively useful, but consider yourself warned.]

Wright and Peeples don’t claim that their methodology is the be-all and end-all of statistical analysis; it’s just a test to see if it’s doable here. So it would be just as foolish to draw any conclusions from their results (beyond the fact that they were able to get results, which was the point), as it would be to jump all over them for leaving out important considerations. Still, with that caveat, let’s do both!

-=-=-=-=-

The methodology was very simple. First, they needed a base number they could use to compare people. They chose the difference between the most commonly imposed guideline sentence for a given arraignment offense, and the actual sentence achieved in a particular case (looking at statewide North Carolina data.) So someone who could negotiate eighty days off a case would be considered more effective than someone who only got the prosecutor to come down thirty days. The guideline took the defendant’s criminal history into account, so that variable was already factored in. And because pleas vary from one criminal offense to another — you might get a lot of months off in a plea to a nonviolent offense, while a violent offense at the same level might not have as much wiggle room — they figured out the standard deviation for each offense.

With those basic numbers, it was easy to see how much an individual lawyer varied from the statewide average, in getting pleas for specific charges. They were able to collect a lot of other data about the lawyers and the cases, which enabled them to run a regression analysis to see which variables actually had any effect on those  numbers.

The biggest variable had nothing to do with the lawyer, the defendant or the offense: it was the local prosecutor’s office. Some offices are hardasses, some aren’t. Hardly a surprise there, but also not exactly something within the lawyer’s control. So not a useful variable.

The next biggest variable was the defendant’s criminal history — something that was supposed to have been controlled for by using the guideline as the reference point. Apparently, repeat offenders are more likely to get more time off their sentence. The authors speculate that it might be because everyone in the system thinks the guidelines are too strict for predicates, but it could also be a flaw in the methodology. Whatever the reason, it’s got nothing to do with the lawyer, and so it’s still not a useful variable.

The only variable that was statistically significant that had anything to do with the defense lawyer himself was the amount of time he’d been practicing. Shiny new lawyers in their first 4 years performed slightly above average. They peaked during years 5 to 9. And then it was all downhill from there. A second peak of effectiveness during years 15 to 19, but dropping again after 20 and plummeting in free-fall after 25.

That’s a counter-intuitive observation if ever there was one. And the authors suggest reasons for it, such as becoming used to the way things are done and so less willing to fight for more. Other reasons might be that older, more experienced attorneys were more likely to handle cases with lower point spreads. Or that they got more results that weren’t quantified here, such as acquittals.

Might as well mention some problems with the methodology at this point. At least those that were apparent from the paper (apologies to the authors for any misunderstanding here).

A big one of course was, as always, the sample. But they did their best with what was publicly available, and frankly they could have done a lot worse.

More critically, they focused on a single metric that is probably not as good an indicator of relative merits as they suspect. The data itself shows that pretty much nothing within the lawyers’ control had any significant effect on the plea results. As a way of distinguishing two lawyers in an organization (or online) who graduated law school roughly the same time, the plea differential is kinda useless.

-=-=-=-=-

This could end right there. The study was to determine whether statistical analysis could be used to meaningfully compare attorneys. The authors say yes, but the data really says no. Certainly, the data demonstrates that there is variation among the outcomes that attorneys get, but the causes appear to have nothing to do with the relative merits of the attorneys. That does not mean that all attorneys are equally good; it simply means that the measurement they chose isn’t useful for comparing them.

And that itself doesn’t mean that statistics can’t be used to measure those differences. It only means that this statistic can’t.

-=-=-=-=-

But there are other problems that ought to be addressed by anyone trying to come up with a similar study (or algorithm).

A huge one was that the study ignored wins. Getting an acquittal at trial, getting the evidence suppressed and the case dismissed, persuading the prosecutor not to file charges in the first place. There are lawyers who plead out every case, but many don’t. Some are really good at making a case go away before it ever happens, but a study like this cannot measure that. Some are really good at winning trials, but ditto.

Maybe “ignored” is too strong a word. Because how do you count what didn’t happen? Still, it is an important consideration for any such approach.

Another problem was in the weighting of cases. The authors did acknowledge that different kinds of offenses get treated differently, and finding the mean plea deal and the standard deviation for each offense was certainly the way to go, but they shouldn’t have stopped there. No two cases are alike. Two may involve the exact same offense, but one is still… worse. The lawyer on the worse case isn’t likely to get as great a deal as the one on the less-serious one. That doesn’t make him a worse lawyer.

Maybe that’s why the young tyros were getting better deals than their elders — they handled the easier cases. It may well be that the more experienced lawyers (in addition to getting more acquittals, dismissals and non-prosecutions) were getting better deals than the younger ones could have gotten in those cases. Or not. We can’t tell from this data, is the point.

Anyway, this post is already longer than it was supposed to be. Statistical ranking of lawyers may well be doable, and it may well be useful. But this study doesn’t really support that conclusion. And those who would try to imitate it for publication or for profit need to understand why.

 

Tags: , , , ,

Get a Trackback link

11 Comments

  1. shg, January 2, 2013:

    I assumed the point of the study was to get people to acknowlege it happened and refer to it in blog posts by offering a provocative yet counterintuitive outcome that would ruffle the feathers of plenty of well-regarded, experienced lawyers.

    See? What do I know?

  2. Nathan, January 2, 2013:

    You give people more credit for thinking things through than I do.

  3. Alex R., January 18, 2013:

    People who already have criminal records have practice being defendants. I expect the first criminal case where one is a suspect involves a big learning curve.

  4. Health, May 11, 2014:

    Along with everything which appears to be developing inside this specific subject matter, all your points of view are very exciting. Even so, I am sorry, but I can not subscribe to your entire idea, all be it refreshing none the less. It would seem to everyone that your comments are actually not completely rationalized and in reality you are yourself not really completely certain of your assertion. In any event I did appreciate looking at it.

  5. ????? ?????, September 30, 2014:

    Win Back The Rights With The Aid Of A Dallas Immigration Lawyer getting the chance to visit United States with a hope to negotiate there forever, who thinks of coming back? You retain waiting to discover an opportunity to take advantage of and get there where the dreams オークリー レーダー become reality but the strict immigration laws keep you away from the typical life right now there and you usually do not find correct means to get a right

  6. gf revenge password, October 3, 2014:

    What you do need, however, is an antisocial and a social personality when it comes
    to email. To get more information on Windows Password Policy and to know more about OS Migration, contact the experts at Specops Software.
    With the above four methods, we will be ease to reset
    Samsung password if we forgot Samsung password.

  7. i know that girl accounts, October 7, 2014:

    You may not be the only one thinking of that password.
    Avoid phishing scams by not logging in from emails you receive.
    There are some establishments that can use their Wi-Fi access point to get private information’s about you.

  8. Digitalplayground password, October 9, 2014:

    Install the one which is compatible with your operating
    system and equipped with all security features. This is a much safer
    practice for passwords safety than leaving them around for anyone to see.

    He accessed her email externally, through her webmail
    account, and sent an inappropriate, and explicit email to all of her contacts, including
    the company clients.

  9. brazzer account, October 9, 2014:

    It is no wonder that accounts and companies are getting hacked through passwords especially when IT keeps making password security more burdensome on the user.
    When you access the Internet through the world wide web it
    is extremely important that you safe guard your personal information. It was Blumenthal who added
    the examples of house keys and diaries to the letter that was cowritten by the pair.

  10. Aundrea, October 13, 2014:

    This could contain codes that happen to be really encrypted Word password recovery.
    Create an alphanumeric version of a term you can remember.

    It is no doubt that creating a BIOS password is important to us.
    Securely stores your information with a strong encryption. Not because passwords are insecure, but because how users choose and manage their passwords.

  11. bangbros password, October 16, 2014:

    By using one of the password finder or child surveillance programs available
    for download on the internet, you can see who your child is chatting with or sending email to and help ensure their safety.
    In the Bit – Locker menu, click Turn on Bit – Locker next to the drive(s) you wish
    to encrypt. 1 Simultaneously press Shift+Ctrl+Del keys
    to permanently erase passwords from your browser including history.

Leave a comment