Bad impact?

As I have discussed before, impact factors are tremendously important in the process of evaluating the quality of scholarship and the career prospects of academics. So it is especially troubling to read this editorial called “Show Me the Data,” published in the Journal of Cell Biology this week by two scientific journal editors and the director of a university press, raising questions about the reliability of those impact factors on which so much depends.

Impact factors are calculated, and marketed, by Thomson Scientific, which was formerly known as the Institute of Scientific Information, or ISI. For many years ISI has published citation indexes, and the impact factors are an outgrowth of those works. When an academic is being assessed for tenure, promotion, grant funding or even a salary bonus, it is a common practice to consult a citation index to determine the influence of that professor’s work and to look at impact factors to see if she is publishing in the most influential places. So Thomson/ISI has tremendous influence in the whole structure of academic hiring and rank. The editorial in JCB takes a look at how firm the foundation for that influence is.

The editorial reports on the authors’ attempts to replicate the calculations that result in specific impact factors. The data they were able to purchase from Thomson Scientific contained what they believed were errors in Thomson’s own reported methodology and did not lead to results consistent with the published impact factors for the three journals involved in the study. When queried, Thomson replied that a different set of data was used to calculate impact factor than that sold from their “research group.” But even when they were reportedly given access to the other data set, the authors were unable to replicate the published findings. There conclusions raise a significant concern: “It became clear that Thomson Scientific could not or (for some yet unexplained reason) would not sell us the data used to calculate their published impact factor… Just as scientists would not accept the findings in a scientific paper without seeing the primary data, so should they not rely on Thomson Scientific’s impact factor, which is based on hidden data.”

By itself this paper does not close the book on the accuracy of impact factors. There are many questions one would like to ask, some of which are unanswered because of restrictions on how Thomson Scientific allows purchasers to use the data that they buy. But there are already many other reasons to question the role of citation rates and impact factors in the promotion and tenure process, such as their inability to account for new modes of disseminating the results of research and scholarship. This article simply strengthens the case for a more multifaceted and qualitative approach to academic evaluation.