This page updates an earlier post on the same subject with the aim of simplifying the process of identifying good academic sources. UCL Libraries provide a more in-depth workshop on evaluating information. This guide is intended for undergraduate and masters students in the fields in which I teach and research, but the underlying considerations have much broader application.
Why does source quality matter?
One vital way to demonstrate the importance of an argument, policy proposal, or research project is to present insights from the existing scholarly literature. Doing this successfully requires that you accurately represent of the underlying academic debate. Sources vary in their quality and influence from those coming from highly reputable journals and publishers to papers from predatory publishers. You need to focus on the former and avoid the latter.
What are predatory publishers?
Predatory publishers profit from author publication fees in return for the promise of quick open-access publishing and have little or no regard to quality control. Their journals often have broad and impressive-sounding names (in some cases legitimate journals have been hijacked). The content can range from plagiarized materials (which may have been inserted by editors – this experiment documents an example of this), findings that have been previously published elsewhere (distinct from a replication study), to fundamentally flawed research.
A number of experiments and stings illustrate the depth of this problem. One of the most vivid examples of this involved a predatory journal’s acceptance of a paper called ‘Get me off your fucking mail list’, the content of which is just those seven words repeated over and over (and in a couple of figures for good measure). Another good example is Daniel Baldassarre’s ‘What’s the deal with birds?’, an enjoyable and unscientific read that received coverage for what it revealed about predatory publishing. Other fun examples include a paper ‘written’ by a pair of Simpsons characters and another linking soil, cancer treatment, and Mars.
Identifying predatory journals and publishers among the population of even just open access publishers can be a challenge. The Directory of Open Access Journals (DOAJ) provides a directory of legitimate open access journals.
You’ll also find that most of the journal articles on your class reading lists will come from a relatively small number of respected publishers: Brill, Cambridge University Press, DeGruyter, Elsevier/ScienceDirect, Oxford University Press, Sage, Springer/Palgrave, Taylor & Francis/Routledge, and Wiley are the most common.
Influence varies among non-predatory sources
To best support your work with the existing academic literature, you should rely on recent and influential sources. Class reading lists are often a good starting point for doing this, but when tasked with looking beyond the reading list, there are a few ways to distinguish the importance of a publication.
Publication Citation Counts
The number of times a paper or book has been cited is one way of measuring its influence. Citation counts depend on a publication’s age, distribution, and journal (or publisher, in the case of books) quality. Citation counts also depend on the underlying sample (for example, all online references to a paper versus references contained only in Web of Science journals).
Recent publications, such as those from the last couple of years, may not have any citations due to limited pre-publication distribution.
There are a lot of different ways to assess a journal’s influence, but Scimago provides an intuitive way of doing this. The Scimago Journal Rank weights a journal’s citations by the reputations of the citing journals; the resulting rankings are then depicted visually by quartile within the field(s) in which a journal is categorized. Each journal’s profile page on Scimagojr.com presents a series of boxes color-coded based on the quartile ranking for each year in each of the fields in which a journal is categorized: these can be green (Q1), yellow (Q2), orange (Q3), and red (Q4) and may differ across disciplines.
The ScimagoJR page for Econometrica, a top economics journal. The boxes are indicated in the large red oval.
Most of your important references should come from Q1 or Q2 journals in relevant fields, as these are the most influential. Specialist journals often fall into the Q2 or Q3 range. In general, you should avoid Q4 journals.
Predatory journals tend to be excluded from the database Scimago draws from, but the following quality journals have not yet been included in that database because they’re relatively new: Global Perspectives, Global Studies Quarterly, Journal of Global Strategic Studies, Journal of Historical Political Economy, Journal of International Business Policy, and Journal of Political Institutions and Political Economy.
Old sources are generally not at the forefront of a contemporary academic debate. More recent work will usually benefit from theoretical advances, more/better data, and more parsimonious analysis (such as the application of more advanced and appropriate methods). Old sources can be useful if you are describing a historical debate or are referring to the seminal contribution on a particular topic. Reference to these should be limited, otherwise you are likely to miss more recent advances in the literature.
Non-Peer Reviewed Sources
Sources that have not gone through peer review may be useful depending on the nature of your project.
Working or discussion papers are most frequently hosted at CEPR, CesIFO, NBER, osf.io, SSRN, and Zenodo, although they may also be found on authors’ and universities’ websites. Be sure to check for a published peer-reviewed version, particularly if the working paper is a few years old. For example, papers distributed by NBER often will have the final publication hidden under ‘other versions’ in Google Scholar.
Law review articles may be useful for providing a description of the legal environment surrounding a subject, but these are not peer-reviewed outlets. HeinOnline provides access to many law reviews, but these are best employed as complementary sources to high-quality social science research.
Think tanks often publish papers on specific policy issues. These may be useful for policy-focused research, but you should pay attention to the think tank’s reputation. Usually Wikipedia can provide a useful indicator of this. Pay attention to indicators of ideological or political affiliation and previous controversies.
General media sources can provide useful descriptive material of current or historical events. Consult Wikipedia’s editorial guidelines for sources for guidance on these.
There are limitations to this guidance, as any experienced researcher is likely to observe. Building an understanding of a subject-matter literature is like music composition: you need to know the basics before you can begin to riff on them.
Version: June 2022