Is citation analysis used consistently across various disciplines as a method to evaluate scholarly journal articles?

surf sunday done
Originally uploaded by iblee....

3rd level concept?
As a third level concept (Information Seeking --> Information Retrieval --> Citation Analysis), we will look at citation analysis from the concept of information seeking and information retrieval. Since citations do matter (Baird, 1994) our focus is on the analysis of citations, where citations relate to the cognitive metaphor (CM). Specifically the purpose is looking at: how consistent as a method is citation analysis used in evaluation? While citation analysis seems only remotely interesting, citations are only rising in importance.

No citations will ever be analyzed without first seeking those citations and then, second, retrieving those citations for analysis. This two step process seems obvious but many barriers within libraries exist to make it very hard without first learning a large of amount of implicit navigation. Wayfinding takes on great significance here.

Across the disciplines
Like the study of sub-atomic particles at CERN particle collider in France, the study of citations is done indirectly. Only after a particle collision are relationships exposed; only after publishing an article can a cognitive-metaphor-architecture be found. In looking at the trails these sub-atomic particles leave behind, quantum physicists are learning about hidden relationships of our universe. This same cognitive metaphor holds true for citations. We will see citation analysis (CA) holds reservoirs of metadata --yet to be mined. Information seeking and information retrieval are only becoming more intimately linked to citation analysis.

In looking at the trails these reference particles leave behind, information scientists learn the soft, non-implicit relationships of a citation universe. Therefore the third level (or sub-sub-) concept: is citation analysis, among various types of scholars used consistently across unique, defined disciplines including information science as a method to evaluate scholarly journal articles? Several items need to be determined first.

Cognitive metaphors "Citation 2.0"
Cognitive metaphors seem to dominate the literature for citation analysis. There is talk of citation profiles, document identification elements, cognitive reasoning behind self-citation, purpose behind citing particular articles, and various other elements including problems with citation analysis. Note: there are very few citation elements representing physical substance or "socialness." Think no "Citation 2.0" as of yet.

An example of a citation element representing something physical is a person's name or the name of a place. Another scholar may counter that a person's name stands for a physical person. In reality, though, a person's name is cognitive metaphor representative of their physical nature. The author did not earn his present nickname as being a physical representative from hell with horns (Leecifer) but the fact that the author may mock orthodox religion a bit too much. Something he has changed out of respect for friends. Pseudonyms are great examples of the cognitive metaphor representing the physicality of some thing. Physical metaphors, while not always readily apparent, may offer insights.

Another example is in physical location. Research shows that physical location can have an effect on whether or not an article is valuable for analysis. Note this effect as the person reviewing the article recognizes a physical location, comments on this physicality and selects an embedded citation as an article to read.

A very clear example is most United States information scientists may preferentially select articles from University of California, over articles from the Universidade Federal do Rio de Janeiro. Physical location then serves as a physical metaphor for an information scientist to determine they may better understand the article's information from a same-language author. But, the cognitive metaphor takes precedence in the field of citation analysis.

Now for CA
Is citation analysis (CA) applied consistency across fields of research. The end/means here, as discussed, also looks at the cognitive metaphor, physical metaphor or some aggregation of both to place and to appropriately relate those concepts. CA seems a whirling dervish of mystery at present, not yet fully studied and is more aptly understood as a cognitive metaphor. Socialness is still hidden.

The Review
Qualitative research can be subjective. Qualitative research also belongs in the cognitive metaphor domain. A quantitative analysis would be representative more of a physical metaphor domain. Has enough citation analysis been done to solidify CA as a methodology? Does CA have a dominant paradigm as a guiding model and for that matter with the information explosion in recent years: has there been a paradigm shift? Has the research on CA been done proving its efficacy as qualitative or at least with enough determinacy to be effective in research use?

The scope here was limited to journal articles that reviewed citation analysis as a theory, as a tool uncovering discrete information about consistency across disciplines, and as a cognitive metaphor or physical metaphor. Since citation analysis itself is meaningless out of context and only in the context of determining a particular "x" variable, it is necessary to take this 3-pointed view. The particular "x" variable we are looking at would be, "Citation Analysis as a Tool in Journal Evaluation" (Garfield, 1978) --can CA actually tell scholars something profound?

As stated before, information seeking and retrieval are immediate nuclear family members of citation analysis. While citation analysis is the newest addition in the family, this scholar's information seeking was relegated to literature from various scholarly journal articles. Other publication types (Blogs) were excluded for the purpose of this post. The articles were not necessarily selected because they were directly in the field of Information Science. Rather the information sought and retrieved were articles that dealt with CA. The scholarly journal articles' subject content ranged from history to medical to economic citation analysis. All articles though were focused through the lens of Information Science. Included in this search and retrieval were also various Information Science journals where seminal articles on CA had been published.

Citation Analysis as Cognitive Metaphor:
Citation analysis means 1) looking at a group of writing to understand the impact this writing has on others scholars, and 2) as a prerequisite means that you must understand how to determine this impact through citations. Eugene Garfield, in an unprecedented, 1972 article titled "Citation Analysis as a Tool in Journal Evaluation" set the stage for serious inquiry into the field of CA. While prior research had been done in looking at citations, very little had been done in looking at the impact or cascading effect that citations may have. Garfield's research question was by reviewing citations can one determine what journals matter the most for research purposes.

The grounds for this research was seen as some articles are repeatedly cited and Garfield sought to confirm the hypothesis that this happened time and time again. Drawing the citations from Science Citation Index, Garfield proved a few journals and more specifically, the articles in those journals account for over half of all journal citations. Much research had been carried out on citations preceding Garfield's.

Garfield's research showed what only previous researchers could allude to: the circle of influence for citations is a small compact loop. Garfield effectively tightened what was a loosely held belief. Clinching up the loop, Garfield showed that many articles may tie off a core loop of journals which taken further meant a core loop of articles could be found in any research area. This was done through the methodology of a cognitive metaphor. This metaphor was the now confirmed cognitive structure of citation analysis. Scholars were shown that the majority of research articles are published in a small set of journals. Largely before this scholars were left to guess.

Garfield's article has been cited over 400 times since its publication. Much of the references pointing to Garfield's article make another point about citation analysis. The impact of citations for various reasons was only growing and is not slowing as more of these reasons are exhumed. Garfield reported that we are far from zero-population growth in world of information and are unlikely to ever be there (Garfield, 1972).

Citation analysis was yet to have a consistently used theory though. Leydesdorff states that "citations support the communication of specialist knowledge ...[and]... citation analysis reflects on citation practices" (Leydesdorff, 1998). Leydesdorff reviews many attempts at theories and incursions on citations, citation analysis, and the elements of relational operations that citations create. In short, Leydesdorff furthers a specific, substratic, notion: much information is hidden in citations and can be mined for valuable information. With this research three themes emerge. CA can be viewed from a I) socio-historical analysis, II) quantitative studies with foci of formalization or generalization, and III) policy or management studies who draw upon elements I, or II (Leydesdorff, 1998).

Furthering this thought to build a cognitive metaphor, research on CA must create a long enough continuity with the problem, apply cognitive framework to Information Science, create research projects that span across disciplines, and in the end develop concepts that are put to use (Kuhlthau, 2005). Pierce metaphorically plunges a shovel down into the information science soil to show that Kuhlthau's idea can be explained through CA. Pierce says CA shows information transfer crosses through three methods; borrowing, collaboration, and boundary crossing are these three. With boundary crossing being the most powerful form of information transfer through which CA can be applied, Pierce showed just how dynamic citations can be (Pierce, 1999). The work of Cameron and also Chudnov, Crooker, and Parker state boundary crossing CA is "so enormous and complex that librarians should cooperate to create one outstanding and highly functional database (Cameron, 1997; Pierce, 1999)" unifying the depth and breadth of CA. Case and Higgins in their review of CA find that while CA is used to study how productive scholars are, and unified databases may prove fruitful, what else is there to citation behaviors; what else can CA tell us about the cognitive relationships hidden in citations (Case, 1991; Case & Higgins, 2000)?

There is another way one must look at CA. If CA is being done consistently, (and here we now see consistent to mean the analysis of citations but we do not mean with the same cognitive focus of that analysis), what are the issues? In a series of retrieved articles, "The Use and Misuse of Citation Analysis in Research Evaluation," by Kostoff (1989), "Problems of Citation Analysis: A Critical Review," by MacRoberts and MacRoberts (1998), "A Compendium of Issues for Citation Analysis," by Phelan (1999), all review the consistent lack of attention paid to the problems of CA. MacRoberts and MacRoberts especially state while critics report upon the acknowledgment of the limitations of CA, they find that little work has been done to resolve this: a specific example they note, as this scholar mentioned earlier, is with scholars' names. What happens when a scholar gets married? How do citation analysts account for this seemingly simple and common social phenomenon? MacRoberts and MacRoberts answer, upon reviewing the literature, is that citation analysts cannot (MacRoberts & MacRoberts, 1989).

With, Garfield, CA was only beginning. Another shock-wave peeling back how far the scholars could go into citations analysis came from the explosion of the world wide web. With the birth of the Internet, new questions were raised about citation behaviors and what could be found from CA. In a paper cited 2,240 times, Kleinberg put forth an algorithmic formulation for the extraction of citation data into meaningful information. The articles impact was indubitably not missed. Research into citations, it showed, could be furthered by even more by heuristics that address other cognitive domains nested in citations. A point of interest here is that Kleinberg put as subject descriptors [Information Seeking], [Information Retrieval], tying citation analysis to these particular cognitive domains (Kleinberg, 1999). As for the future state of CA, Perkel (2005) reports that services like google scholar will ultimately mean a much deeper mining of data to include document elements beyond reference lists such as acknowledgments and that a paper like Kleinberg's will only be drawn upon even more for citation analysis (Perkel, 2005).