Let's dumb-up (journal citation) impact factors

The following is a commentary on an editorial in the British Medical Journal entitled: Let's dump impact factors Kamran Abbasi, acting editor BMJ 2004;329 (16 October), doi:10.1136/bmj.329.7471.0-h http://bmj.bmjjournals.com/cgi/content/full/329/7471/0-h I've submitted the following commentary. It should appear Monday at: http://bmj.bmjjournals.com/cgi/eletters/329/7471/0-h Prior Amsci Topic Thread: "Citation and Rejection Statistics for Eprints and Ejournals" http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/0138.html http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/1112.html ---------------------------------------------------------------------- Enrich Impact Measures Through Open Access Analysis (Or: "Let's Dumb-Up Impact Factors") Stevan Harnad The "journal impact factor" -- the average number of citations received by the articles in a journal -- is not an *invalid* instrument, but a blunt (and obsolescent) one. It does have some meaning and some predictive value (i.e., it is not merely a circular "definition" of impact), but we can do far better. Research impact evaluation should be thought of in multiple-regression terms: The journal impact factor is just one of many potential predictive factors, each with its own weight, and each adding a certain amount to the accuracy of the prediction/evaluation. The journal impact factor is the first of the regression weights, but not because it is the biggest or strongest, but just because it came first in time: Gene Garfield (1955, 1999) and the Institute for Scientific Information (ISI) started to count citations (and citation immediacy, and other data) and produced an index of the average (2-year) citation counts of journals -- as well as the individual citation counts of articles and authors. The fact that unenterprising and unreflecting evaluation committees found it easier to simply weight their researchers' publication counts with the impact factors of the journals in which they appeared was due in equal parts to laziness and to the valid observation that journal impact factors *do* correlate, even if weakly, with journals' rejection rates, hence with the rigour of their peer review, and hence with the quality of their contents: "High citation rates... and low manuscript acceptance rates... appear to be predictive of higher methodological quality scores for journal articles" (Lee et al. 2002) "The majority of the manuscripts that were rejected... were eventually published... in specialty journals with lower impact factor..." (Ray et al. 2000) "perceived quality ratings of the journals are positively correlated with citation impact factors... and negatively correlated with acceptance rate." (Donohue & Fox 2000) "There was a high correlation between the rejection rate and the impact factor" (Yamasaki 1995) But even then, the article and author exact citation counts could have been added to the regression equation -- yet only lately are evaluation committees beginning to do this. Why? Again, laziness and unenterprisingness, but also effort and cost: An institution needs to be subscribed to ISI's citation databases and needs to take the trouble to consult them systematically. But other measures -- richer and more diverse ones -- are developing, and with them the possibility of ever more powerful, accurate and equitable assessment and prediction of research performance and impact (Harnad et al. 2004). These measures (e.g. citebase http://citebase.eprints.org/) include: citation counts for article, author, and journal; download counts for article, author and journal; co-citation counts (who is jointly cited with whom?); eventually co-download counts (what is being downloaded with what?); analogs of google's "page-rank" algorithm (recursively weighting citations by the weight of the citing work); "hub/authority" analysis (much-cited vs. much-citing works); co-text "semantic" analysis (what -- and whose -- text patterns resemble the cited work?); early-days download/citation correlations (http://citebase.eprints.org/analysis/correlation.php) (downloads today predict citations citations in two years (Harnad & Brody 2004); time-series analyses; and much more. So the ISI journal-impact factor is merely a tiny dumbed-down portion of the rich emerging spectrum of objective impact indicators; it now needs to be dumbed-up, not dumped! Two things need to be kept in mind in making pronouncements about the use of such performance indicators: (i) Consider the alternative! The reason we resort to objective measures at all is that reading and evaluating every single work anew each time it needs to be evaluated is not only subjective but labour-intensive, and requires at least the level of expertise and scrutiny that (one hopes!) the journal peer review itself has accorded the work once already, in a world in which qualified refereeing time is an increasingly scarce, freely-given resource, stolen from researchers' own precious research time. Citations (and downloads) indicate that researchers have found the work in question useful in their own research. (ii) The many new forms of impact analysis can now be done automatically, without having to rely on ISI -- if and when researchers make all their journal articles Open Access, by self-archiving them in OAI compliant Eprint Archives on the Web. Remarkable new scientometric engines are just waiting for that open database to be provided in order to add the rich new panoply of impact measures promised above (Harnad et al. 2003). Donohue JM, Fox JB (2000) A multi-method evaluation of journals in the decision and management sciences by US academics. OMEGA-INTERNATIONAL JOURNAL OF MANAGEMENT SCIENCE 28 (1): 17-36 Garfield, E., (1955) Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas. SCIENCE 122: 108-111 http://www.garfield.library.upenn.edu/papers/science_v122(3159)p108y1955.htm Garfield E. (1999) Journal impact factor: a brief review. CMAJ 161(8): 979-80. http://www.cmaj.ca/cgi/content/full/161/8/979 Harnad, S. and Brody, T. (2004) Prior evidence that downloads predict citations BMJ Rapid Responses, 6 September 2004 http://bmj.bmjjournals.com/cgi/eletters/329/7465/546#73000 Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S., Gingras, Y, Oppenheim, C., Stamerjohanns, H., & Hilf, E. (2004) The Access/Impact Problem and the Green and Gold Roads to Open Access. SERIALS REVIEW 30. http://www.ecs.soton.ac.uk/~harnad/Temp/impact.html Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. ARIADNE 35 (April 2003). http://www.ariadne.ac.uk/issue35/harnad/ Lee KP, Schotland M, Bacchetti P, Bero LA (2002) Association of journal quality indicators with methodological quality of clinical research articles. AMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 287 (21): 2805-2808 Ray J, Berkwits M, Davidoff, F (2000) The fate of manuscripts rejected by a general medical journal. AMERICAN JOURNAL OF MEDICINE 109 (2): 131-135. Yamazaki, S (1995) Refereeing System of 29 Life-Science Journals Preferred by Japanese Scientists SCIENTOMETRICS 33 (1): 123-129