Over at Balkinzation and Freakonomics under the banner "Iatrogenic Legal Assistance?" Ian Ayres refers us to a study bound to give a certain subset of lawyers and law professors pause:

Harvard Professors Jim Greiner and Cassandra Pattanayak have posted a remarkable randomized experiment (“What Difference Representation?”) with evidence showing that offers for free legal representation from the Harvard Legal Aid Bureau (HLAB) ended up hurting unemployment claimants.

The claimants who were offered representation were no more (or less) likely to win their administrative appeal – but “the offer caused a delay in the proceeding.”  The claimants offered representation had to wait on average 42 percent longer (53.1 vs. 37.3 days) before they received a decision of an Administrative Law Judge.

The results are particularly striking because not everyone who was offered representation was represented, and because those who were not offered HLAB representation were sometimes represented by alternative organizations.

I’m no stranger to concept of iatrogenic medicine and I think it’s a widespread problem, having seen scores of examples of it in my practice. My Greek is rusty, but I believe iatrogenic = iatros (healer) + genic (created by) = "created by a healer." We’re talking about a problem pertaining to the law or lawyers, or nomikos, so it’s really nomikogenic injury, and I’m sure there’s plenty of that, too. Lawyers commit their fair share of malpractice.

But this study asks something different; the authors didn’t think that poor representation was at issue, given their "personal observations of the extraordinarily talented set of students HLAB recruits, the students’ dedication, the tremendous oversight and guidance the clinical staff at HLAB provides, and what we understand to be the strong reputation HLAB enjoys for its work in this area."

Which means the authors examined legal representation when it works how it’s supposed to work. When it works how it’s supposed to work, it doesn’t work, it just prolongs the case.

That’s a problem.

As Ayres continues:

The study highlights, again, the simple power of randomized control studies.  There is a persuasive transparency to randomized control trials.  The randomization doesn’t tell us why the offers caused a delay, but we should be fairly confident that those who were lucky enough not to be offered free legal assistance by HLAB had a better shot of cashing unemployment checks sooner.  This initial study’s main limitation is that its sample size is only 207.  Still, that is sufficient to raise the serious concern that HLAB’s offers of representation are hurting its potential clients. In medicine, iatrogenic effects are adverse side effects caused by medical treatment – this study points to a legal analogue in which well-intentioned legal assistance ends up resulting in adverse “side effects” for the clients.

And that’s where things fall about. "Simple" is only a stone’s throw from "simplistic." I’m certainly guilty of using language more carelessly on this blog than I do in practice, and I’m sure Ayers does the same on Balkinization and Freakonomics, but it’s frustrating to see a "a professor of law and economics at Yale" on two popular blogs talk about the "persuasive transparency" of randomized control trials, as if he wasn’t aware of how prone such things are to misinterpretation. The truth underlying "simple" and "transparent" answers provided by trials is often much more nuanced and complicated than a brief comparison of the numbers would suggest.

Like all good scientists, the actual authors of the study devoted one-third of the study to examining what may have been wrong with their methodology and what could be improved in the future. Two issues jump out at me, neither of which relates to sample size (which was, indeed, a problem).

First, the study focused on administrative appeals of unemployment compensation claims. I don’t mean to belittle the importance of these claims — which are how thousands of people put food on the tables — or to besmirch the efforts of HLAB, but there’s not much about these claims that requires legal representation. There are no "legal" disputes in unemployment claims (everyone agrees on the applicable law) and, by the time you get to an administrative appeal before an Administrative Law Judge, a neutral claims adjuster paid by the state has already requested the most pertinent documents and interviewed the parties.

That means that the study looked at a unique procedure in the law that doesn’t even involve three of the most important things that civil lawyers do:

  • analyze legal issues (and then advise the client accordingly);
  • develop legal arguments (and then make them persuasive for the court); and,
  • investigate the circumstances and obtain evidence that supports your client’s claims.

Indeed, that’s part of why we let law students do unemployment appeals work: because it doesn’t require a full-fledged, licensed attorney. If you’re no regular Cicero, you might benefit from an advocate, but you don’t need a lawyer.

I can only think of two analogous situations: traffic court and small claims court. In both of those, it’s my sense that a consensus of lawyers would say "sure, you can have a lawyer, but you don’t need one." There are no legal issues to unravel or to strategize around, no need to condense decades of precedence and hundreds of pages of statutes and regulations into a coherent brief, and minimal need for discovery. Even if you could depose witnesses, who would you depose? The cop who wrote you the ticket? We know his or her side of the story: it’s written on the ticket.

All of which is to say, the authors may have proven that you don’t need a lawyer to represent you in simple administrative agency hearings — something with which many, perhaps most, lawyers would agree — but that’s not quite like showing that civil lawyers in general, or legal aid lawyers in particular, are useless.

Second, if we’re going to put our faith in "the simple power of randomized control studies," then we need to remember the many simple ways that randomized control studies go wrong, like selection bias.

Consider this nugget from the study: 

[T]he win rate for the control group in our study (those not offered HLAB representation) was higher than the average success rate in Massachusetts first-level appeals as a whole. In our dataset’s control group, when the claimant appealed, she prevailed 65% of the time; in the overall Massachusetts system, the win rate for appealing claimants was 47%. In our dataset’s control group, when the claimant defended against an employer appeal, she prevailed 83% of the time, compared to a 75% rate in Massachusetts overall.

That is to say, the folks who called HLAB after losing before the claims adjuster were almost forty percent more likely win their cases than your typical claimant, and the folks who called HLAB after winning before the claims adjuster were almost ten percent more likely win their cases again than your typical claimant. That’s a huge difference.

The authors come up with a hypothesis:

[T]he disparity in the win rates, particularly for claimant-initiated appeals, is striking. This suggests to us either that the claimants who initiated contact with HLAB possessed personal characteristics making them more likely to win their cases or that the underlying facts in their cases were unusually strong. It is not immediately apparent to us why claimants with cases characterized by strong facts would be unusually likely to call HLAB. But when we note that the HLAB unemployment intake system depended on a telephone contact initiated by the claimant, it appears plausible to us to suppose that the claimants who initiated contact with HLAB were perhaps more persistent, articulate, and motivated than the average claimant. These are skills that would serve a pro se claimant well in litigating a first-level appeal.

Indeed they would. The whole situation also reminds me of when Judge Morris B. Hoffman found, along with economists Paul Rubin and Joanna Shepherd, that private criminal defense attorneys had a better record than public defenders because "marginally indigent" defendants choose to retain a private criminal defense attorney based upon "a combination of the seriousness of the offense and the likelihood of conviction." As Judge Hoffman wrote:

Imagine a guilty, marginally indigent defendant facing a relatively minor felony (for which he will most likely get probation). Now add to the mix the fact that his crime was captured on videotape, meaning he has a small chance of avoiding conviction. It is unlikely such a defendant would deplete his and his family’s and friends’ resources to hire a private lawyer when he could get a free public defender to achieve the same result.

At the other end of the spectrum, imagine a marginally indigent defendant charged with first degree murder, and imagine that he is innocent. Wouldn’t that defendant do everything in his power to marshal the resources to hire a private lawyer, if he believed, rightly or wrongly, that the private lawyer were more likely to achieve an acquittal?

In other words, marginally indigent defendants who choose public defenders tend to be guilty. And of course if that’s true, it’s not at all surprising that public defenders would achieve less favorable outcomes.

I imagine a similar dynamic is at work in the unemployment claims. What kind of a claimant goes through the effort to contact HLAB? The type of claimant who either believes strongly in their own case, or the one who has a large claim, or a mixture of both. People who find their own claims unworthy of the time it takes to talk to HLAB are likely the ones who have the weakest claims or, at least, are the people who are least willing to devote substantial time and energy to their claims.

Thus, the HLAB study may have been sampling Lake Woebegone: most of the HLAB callers have stronger than normal claims, and are more persistent than normal people, and so are the least likely to benefit from representation.

Might this study mean that HLAB should shift its focus to proceedings where its students are more likely to have some impact? It might, but, at the same time, I would be curious about what would happen if HLAB disclosed to its potential clients the results of the study. Being your own lawyer, even in a court where the law is simple and transparent like in unemployment appeals, is stressful and time-consuming. I’d bet a number of the clients would be willing to take a potential two-week delay on their claim in order to ensure everything was done by the book.