Shepardizing Science: Is an Article Fact or Fiction?

Ken Strutin, director of legal information services at the New York State Defenders Association, has an article in the September 27th issue of the New York Law Journal, “Shepardizing Science: Is an Article Fact of Fiction?,” calling for a need to create “forensic bibliometrics” tools, similar to law citators.

The author points out that “In the scientific publishing lexicon, three levels of caution, which resemble Shepard’s signals, are the most salient: Retraction, Expression of Concern and Correction.”

From the article:

   It was Frank Shepard’s methology that paved the way for Eugene Garfield’s creation of the Science Citation Index (SCI), and ultimately, the page ranking protocols used by Internet search engines. [footnote omitted]  Most legal opinions can be Shepardized, and along with a full court press of bibliometric analysis in multiple sources, this tool can provide a high level of quality assurance.  The same is not easily accomplished in the scientific disciplines.

   Aside from the tools already noted, quality control of scholarly literature would benefit from something resembling a Shepard’s for scientific research.  It would be a universal mechanism that flags retracted articles in peer review journals and treatises, in all formats and at all access points, clearing indicating which ones should not be cited or relied upon.

The article clearly sets out the reasons why “. . . an expert in the citation analysis of scientific literature can play a crucial role in litigation.”

Distinguishing Judges: An Empirical Ranking of Judicial Quality in the United States Courts of Appeal

I just happened across this new article in the Spring 2011 issue of the Missouri Law Review, “Distinguishing Judges: An Empirical Ranking of Judicial Quality in the United States Courts of Appeal,” by Robert Anderson IV.

The abstract reads:

“This article presents an empirical performance ranking of 383 federal appellate judges who served on the United States Courts of Appeals between 1960 and 2008.  Like existing judge evaluation studies, this Article uses citations from judicial opinions to assess judicial quality.  Unlike existing citation studies, which treat positive and negative citations alike, this Article ranks judges according to the mix of positive and negative citations to the opinions, rather than the number of citations to those opinions.  By distinguishing between positive and negative citations, this approach avoids ranking judges higher for citations even when the judges are being cited negatively.  The results are strikingly different from those found in the existing citation count-based studies of judicial performance.  When the mix of positive and negative citations is taken into account, many of the most highly cited judges from the citation-count studies are only average and some of the average judges in the citation-count studies emerge as the most positively cited.  The results suggest there is an objective performance measure that can measure judicial performance and provide incentives for fidelity to the rule of law.”


Citations to Foreign Law in the Supreme Court of Canada

Waiting for Globalization: an Empirical Study of the McLachlin Court’s Foreign Judicial Citations

Peter McCormick

41 Ottawa Law Review 209

From the abstract:

…This paper explores the Supreme Court of Canada’s citations to judicial authority since 2000. The paper argues that the notion of non-Canadian citation must be disaggregated into three component parts – English, American and everything else. – before it can usefully be examined, these three exhibiting quite different patterns; an d its concludes that in none of them can the “expanding globalization” thesis be sustained. … Finally, it looks at the kinds of cases that tend to include non-Canadian citations, and suggests that not only are we still waiting for globalization, but to the extent that we are focusing primarily on rights-based jurisprudence, we may also be looking in the wrong place.

Citation Advantage of Open Access Legal Scholarship

“Citation Advantage of Open Access Legal Scholarship”

UGA Legal Studies Research Paper No. 11-07

JAMES M. DONOVAN, University of Kentucky College of Law Library
CAROL A. WATSON, University of Georgia Law School

To date, there have been no studies focusing exclusively on the impact of open access on legal scholarship. We examine open access articles from three journals at the University of Georgia School of Law and confirm that legal scholarship freely available via open access improves an article’s research impact. Open access legal scholarship –  which today appears to account for almost half of the output of law faculties – can expect to receive 50% more citations than non-open access writings of similar age from the same venue.

Bloomberg Law’s discounts challenge information suppliers

“Bloomberg Law’s discounts challenge information suppliers” is the headline to a story in today’s Financial Times (p. 19) by Andrew Edgecliffe-Johnson.

The story quotes Lou Andreozzi, the new head of Bloomberg Law, on the company’s efforts to persuade attorneys to consider at least replacing one of their “Wexis” accounts with Bloomberg Law, since Bloomberg’s flat rate pricing (quoted at $ 450 per attorney) is preferable to the “more expensive and unpredictable sums” charged by the competition.

The story also reports how “Bloomberg has recruited ‘hundreds’ of lawyers to create a citation system, which advises users whether cases are still in use, to rival those owned by Westlaw and Lexisnexis.”

The story quotes analyst David Curie who says that “Bloomberg looked unlikely to make big inroads in the short term, but its ‘deep pockets’ made it a long-term challenger.  ‘The pricing definitely is the most challenging and disruptive thing about it,’ he said, predicting that others may follow its flat fees . . .

The story  includes a sidebar, “Legal services industry continues to expand” which includes this information:

Law firms and corporate legal departments once looked to legal research services for basic case law, newspaper articles and public records.

As such information has become more freely available, companies such as Thomson Reuters’ Westlaw division and Reed Elsevier’s LexisNexis have concentrated to turning their databases into online tools to enhance clients’ productivity.

The sidebar goes on to use Thomson’s acquisition of Pangea3 as an example.

The Cost of Judicial Citation: An Empirical Investigation of Citation Practices in the Federal Appellate Courts

From the just-received Volume 2010, Issue 1, Spring University of Illinois Journal of Law, Technology & Policy, at page 51:

The Cost of Judicial Citation: An Empirical Investigation of Citation Practices in the Federal Appellate Courts

by Casey R. Fronk


Since the early 1960s, computerized legal research technology has enabled judges and their law clerks to access legal information quickly and comprehensively. Particularly for appellate judges, who rely on wide-ranging legal research when writing opinions, this technological change has had special resonance. This Article attempts to quantify the effects of computer- assisted legal research on the federal judiciary by empirically analyzing citation patterns over the past fifty years. The results of this analysis suggest that the digitization of legal research has had statistically significant effects on the amount and style of citation in judicial opinions. Although the average number of cases cited in opinions has doubled between 1957 and 2007, the number of cases cited only in string citations has decreased by nearly the same percentage. This Article argues that such results can be explained by a basic economic theory of judicial citation in which judges respond to the decreasing cost of opinion production by discarding string citation for more effective communicative techniques.


This Article proposes that a simple microeconomic approach can describe judicial citation practices over the last fifty years.  It provides empirical evidence that judges use citations in part as a communication device, and that the cost of legal research is intimately connected with the effectiveness of this communication (and therefore with judicial citation patterns).  The empirical results in this Article not only demonstrate the effectiveness of the microeconomic approach in describing  judicial opinion style, but also provide a foundation for future research into the effects of judicial ideology on citation practices.

Properties of the United States Code Citation Network

“Properties of the United States Code Citation Network”

MICHAEL JAMES BOMMARITO, University of Michigan at Ann Arbor – Department of Political Science, University of Michigan at Ann Arbor – Department of Mathematics, University of Michigan at Ann Arbor – Center for Study of Complex Systems

DANIEL MARTIN KATZ, University of Michigan Law School , University of Michigan at Ann Arbor – Center for Study of Complex Systems, University of Michigan – Department of Political Science

The United States Code is a body of documents that collectively comprises the statutory law of the United States. In this short paper, we investigate the properties of the network of citations contained within the Code – most notably its degree distribution. Acknowledging the text contained within each of the Code’s section nodes, we adjust our interpretation of the nodes to control for section length. Though we find a number of interesting properties in these degree distributions, we demonstrate that a power law distribution is not an appropriate model for this system.


Source:  LSN Experimental & Empirical Studies Vol. 10 No. 103,  12/02/2009

Using Citation Analysis Techniques for Computer-Assisted Legal Research in Continental Jurisdictions

“Using Citation Analysis Techniques for Computer-Assisted Legal Research in Continental Jurisdictions”

ANTON GEIST, University of Edinburgh – School of Law, University of Vienna – Faculty of Law

The following research investigates the use of citation analysis techniques for relevance ranking in computer-assisted legal research systems. Overviews on information retrieval, legal research, computer-assisted legal research (CALR), and the role of citations in legal research enable the formulation of a proposition: Relevance ranking in contemporary CALR systems could profit from the use of citation analysis techniques. After examining potential previous work in the areas of Web search, legal network analysis, and legal citation analysis, the proposition is further developed into a testable hypothesis: A basic citation-based algorithm, despite all its shortcomings, could be used to significantly improve relevance ranking in computer-assisted legal research. By computing and analysing the distribution of 242,078 headnote citations across 80,195 opinions written by the Austrian Supreme Court of Justice between 1985 and 2008, proof for this hypothesis is presented.

Source: LSN Legal Education Vol. 6 No. 25,  06/09/2009