What makes a journal valuable?

For almost 90 years, librarians, faculty authors, tenure review committees and publishers themselves have relied on a single measure – the impact factor – to determine the relative quality of different scholarly journals. Impact factors are based on the number of times articles from a particular journal are cited in other scholarly articles. The citations to articles in one journal are cumulated to calculate the impact factor. It is fairly obvious that this system has some problems, however. For one thing, frequency of citation is a poor marker for quality, since all cited references to a work may not be positive and approving. To posit an extreme example, many articles that cite one specific article as a particularly bad example will boost the citation rate for that article and could raise the impact factor of the journal that published the flawed study. Also, journals are not all of equal quality or influence (which is the point, after all), so many citation from peripheral journals may not be as important as one or two citations in the really influential and universally-read publications. Impact factor can flatten these distinctions in regard to a single article, although cumulation over time should cause the “best” journals to rise to the top.

A new measure of journal quality, called the Eigenfactor, tries to address this last problem by starting with an evaluation of journal quality and assessing article impact on that basis. As their explanation of their methods says,

“Eigenfactor provides a measure of the total influence that a journal
provides, rather than a measure of influence per article… To make our
results comparable to impact factor, we need to divide the journal
influence by the number of articles published.”

Leaving aside the complex mathematics explained at their site, the Eigenfactor is based on an algorithm that maps how a hypothetical researcher would move from article to article based on cited references. This mapping yields a measure of the amount of time that researcher would spend with each particular journal. The score of a journal is based on that finding, and the influence of articles is measured by the influence of the journal in which they are published. This method corrects for peripheral citations and, it is claimed, for different citation patterns in different disciplines.

Both of these methods, however, measure the quality of journals only from within the relatively closed world of traditional periodical publication. Can we imagine ways of assess journal quality that can account for external factors and hence for the changes that are occurring within scholarship?

The advent of online aggregators of journal content has offered one relatively simple external measurement of journal impact which librarians have been quick to embrace – cost per article download. It used to be very cumbersome to try and tally which print journals were most used in a library, based on how often copies were picked up and reshelved. Now databases offer constantly updated counts of downloads which are easily divided into the cost of the database to provide a measure of where collections budgets are best spent. Since many downloads will reduce the cost per download, this metric also can serve as a rough indication of quality, or, at least, influence.

The real question I have, however, is how to assess the importance of traditional journal publication vis-à-vis newer, informal means of communication that are growing in importance amongst scholars. As blogs, wikis and exchanges of working papers via e-mail grow, scholars are getting their inputs and influences from new sources, and web publication of various kinds often supplements, and occasionally supplants, tradition publication. As the ACRL’s recent paper on “Establishing a Research Agenda for Scholarly Communications” puts it,

“Extant measures may suffer from being tightly coupled to traditional
processes while also inhibiting the application of other measures of
value. In the new digital environment, activities other than traditional
or formal publication should be valued in the reward structure for scholarship.”

I know of no metric that can yet account for the variety of informal publications and their relative influence. That, of course, is why it is part of a research agenda. As these informal, digital means of sharing scholarly work become more common, one of the principle functions of traditional publication – that of communicating the finished products of research – may become less and less important. Other functions, such as registration, certification and preservation, may continue to rely on traditional journals for a longer time. But the academic world needs to look carefully for ways to evaluate and compare the influence of a variety of new communications if it is to value scholarship based on its true impact.