Scholarly Quality Assessment for Research in Econometrics

The SQARE initiative aims to introduce a novel quality assessment system for research articles in Econometrics, offering an alternative to the conventional quality signal provided by academic journals. We envision establishing an online platform akin to a journal where authors can submit their papers for evaluation by referees and editors. However, rather than publishing the articles themselves, our focus will be purely on determining and disseminating quality assessments of each submission. This eliminates the need for a simplistic accept/reject decision on manuscripts. It also enables us to immediately compete with leading journals, because authors can submit their work to SQARE concurrently with journal submissions. Our goal is to modernize the existing academic evaluation framework by providing a more efficient, nuanced, and collaborative research quality assessment.

Background

Aspects of the following discussion may be equally relevant to other academic disciplines, both within and beyond Economics, but our focus here is on Econometrics, in line with the initial objective of the SQARE initiative.

Academic journals have been instrumental in the development and dissemination of Econometric research for many decades. Prestigious journals such as The Review of Economics and Statistics (established in 1919), Econometrica (1933), The Review of Economic Studies (1933), the Journal of Econometrics (1973), the Journal of Time Series Analysis (1980), Econometric Reviews (1982), the Journal of Business & Economic Statistics (1983), Econometric Theory (1985), the Journal of Applied Econometrics (1986), and the Econometrics Journal (1998), among others, have played pivotal roles in shaping the field. Many of our academic role models have played an integral role in founding and editing these journals, and our appreciation for the historic role of these journals and the dedicated individuals who have steered them through the years is completely separate from any debate on the continuing relevance of the existing academic journal model for the future.

The academic journal, as we recognize it today, was fully developed within the 20th century, well before the advent of the internet, email, or the World Wide Web. These journals have historically fulfilled several functions:

• Research Dissemination
• Preservation and Archiving
• Establishment of Citable References
• Quality Assurance and Evaluation through Peer Review

In the digital age of the 21st century, platforms like arXiv have increasingly assumed responsibilities for research dissemination, preservation, archiving, and establishing citable references, significantly diminishing the need for academic journals to perform these roles. Nowadays, the primary function of Econometric journals lies in quality assurance — they serve as a robust indicator of quality, highly regarded in academic assessments across various domains. However, we are concerned that this journal publication system is becoming increasingly inefficient for two main reasons:

1. While essential for conventional journals, the traditional accept/reject decision model proves increasingly inadequate for authors and reviewers alike. Within today’s highly competitive academic landscape, authors frequently face the need to sequentially submit their manuscripts to multiple journals before achieving publication. A modern quality assessment system would bypass this binary decision, offering a more nuanced and efficient evaluation approach.
2. Despite the utilization of internet platforms and email communication, the editorial and publication processes of academic journals remain largely unchanged from the days of postal correspondence. This persistence in old methodologies fails to leverage the advantages offered by modern web platforms, which could substantially improve the experience for all parties involved and introduce a more collaborative decision-making process not solely reliant on individual editors.

Although there have been attempts to move away from the binary accept/reject model (e.g., the establishment of Berkeley Electronic Press in 1999 or the practice of publishing some rejected Econometrica articles in QE), the fundamental issue (1) seems structurally ingrained within the traditional journal framework. While modernizing the editorial process (2) is theoretically feasible within existing journals, significant innovation is unlikely to stem from these institutions due to the plethora of stakeholders involved (editors, trustees, associations, publishers, etc.) that would need to agree on such changes.

If this analysis is accurate, it suggests that real change and innovation in the academic evaluation system are unlikely to come from journals, whether existing or newly created. Instead, we propose that significant advancements could arise from the establishment of quality assessment systems dedicated exclusively to quality assurance. These systems would leave the tasks of dissemination, preservation, and reference establishment to existing platforms like the Econometrics arXiv.

Several platforms are already reshaping traditional academic publishing and feedback mechanisms. For instance, ‘The Unjournal’ and the ‘Northwestern Research Feedback Project’ introduce innovative methods to enhance research through various stages of feedback and evaluation. However, these differ significantly from our proposal. Most crucially, SQARE is dedicated exclusively to assessing research quality within the field of Econometrics.

A potential challenge in transitioning from traditional academic journals to a pure quality evaluation system might be the financial model. Academic journals currently depend on publishing articles for revenue, whereas a system focused solely on quality assessment would not publish articles, affecting its revenue stream. However, the traditional journal subscription model is already waning, and though the open-access model seems more viable, open-access publication fees essentially just redirect funds that could easily support research directly. In any case, the financial aspect, while worthy of discussion, may not present a substantial obstacle to innovation, because the bulk of editorial and refereeing work is already performed voluntarily today, strongly suggesting that financial constraints will not be prohibitive to change.

The SQARE score

Given our exclusive focus on quality assessment, we can move beyond the conventional binary decision of acceptance or rejection. We propose to evaluate papers on a “five-SQARE scale” (♦ to ♦♦♦♦♦), guaranteeing that every technically correct and readable submitted paper receives a definitive assessment. This approach significantly streamlines the quality assessment process, addressing the current system’s inefficiencies where authors repeatedly submit their work to multiple journals, duplicating refereeing and editorial efforts and increasing publication lags.

In our system, a three-SQARE score (♦♦♦) aims to reflect the quality of leading field journals such as the Journal of Econometrics. A five-SQARE score (♦♦♦♦♦) is designed to correspond to top-tier general interest journals like Econometrica or The Review of Economic Studies. Assigning, communicating, and archiving such a SQARE score for every paper submitted will be the primary goal of the new evaluation system that we propose.

We anticipate that the SQARE score will serve as a vital indicator of research quality, equivalent to the role of journal publications today. This metric will be beneficial for authors themselves to gain independent feedback on how their work is received, particularly for informing decisions on their future research endeavours. Moreover, for individuals other than the authors, thoroughly reading and effectively assessing a paper can be challenging. Consequently, a dependable external quality signal becomes crucial for evaluating publication records in contexts such as hiring, promotions, grant applications, and other professional evaluations. Finally, akin to how journals circulate newsletters and social media updates about newly published or accepted papers, we can similarly disseminate updates on newly evaluated papers.

During a transition period, until the SQARE score is well-established, we anticipate that authors will continue submitting to both SQARE and traditional journals. Once SQARE earns a reliable reputation, we expect that it—and potentially other similar systems—could largely render traditional journal submissions obsolete, thereby fully realizing the efficiency gains for the academic community. The duration and extent of this transition are, of course, difficult to predict. Alternatively, should academic journals be compelled by competition to modernize and address the described inefficiencies themselves, this too could represent a successful realization of our initiative’s goals.

First-round evaluation

Upon submission to SQARE, each paper undergoes an initial evaluation that mirrors the process of one refereeing round at top field journals such as the Journal of Econometrics or the Journal of Business & Economic Statistics. This involves an editor, associate editor, and referees in their conventional roles, but with notable modifications to enhance the process.

Possible first-round outcomes:

0. Reject
1. One SQARE (♦)
2. Two SQARE (♦♦)
3. Three SQARE or higher (♦♦♦, ♦♦♦♦, ♦♦♦♦♦)

Outcomes (0), (1), and (2) conclude the evaluation process for a paper (though authors may choose to withdraw their paper before the evaluation is announced publicly). Outcome (3) offers authors a choice: accept a three SQARE score (♦♦♦) or proceed to a second-round evaluation for a potentially higher rating.

Streamlined refereeing:

There will be one single round of refereeing, and it is not designed to mandate significant revisions or request extensive rewrites from authors, except for the correction of identified mistakes or major problems. It’s important to note, however, that the clarity and presentation of a paper will naturally influence its quality score. Authors are, therefore, encouraged to submit papers that are polished and finalized. This approach is also a strategic choice, acknowledging that, at least initially, our evaluation system will operate alongside traditional journals, that is, authors are likely to submit their papers to both platforms and should not be burdened with conflicting demands for substantial revisions from multiple sources.

Innovative features:

Possible improvements to the standard evaluation process include facilitating communication between anonymous referees and authors within a single review cycle and allowing associate editors to select or indicate their preferences for which manuscript to handle. Using the full flexibility of modern internet platforms should markedly improve the editorial process efficiency.

Once a paper clears the first-round evaluation without being rejected, it is deemed technically sound and free of significant issues, concluding the responsibilities of the associate editor and referees—even if the paper receives an outcome of (3) and the authors opt for a second-round evaluation. Of course, peer review is never perfect, but a single round of refereeing, augmented by improved communication channels between authors and referees during this phase, should be sufficient to uphold the conventional standard of scrutiny.

Second-round evaluation

The first-round evaluation phase outlined above incorporates several forward-thinking elements but remains fundamentally similar to the procedures of current academic journals. We have confidence in this process’s capability to accurately assign one, two, and three SQARE scores (♦, ♦♦, ♦♦♦), based on the expertise of many competent associate editors within the field — these associate editors, with the assistance of skilled referees and under the management of a few main editors, are usually well-equipped to make these evaluations effectively.

However, distinguishing between a three, four, or five SQARE (♦♦♦, ♦♦♦♦, ♦♦♦♦♦) manuscript presents a more complex and subjective challenge, justifying a larger departure from traditional journal practices. For these higher distinctions, we propose a second-round evaluation that draws on the collective wisdom of a broader assembly, such as an editorial board or an “editorial college”, to foster a more collaborative and inclusive decision-making process. This could take the form of a simple discussion and voting mechanism, with the median vote (or some other quantile) determining the final score. Notably, the online publishing platform ‘Scipost’ already successfully employs community engagement and voting in its editorial process, underscoring the practicality and benefits of such an approach.

Should authors opt for a second-round evaluation, having received the highest possible rating in the first round, they would not only have the opportunity to submit a revised version of their paper but would also be asked to provide additional materials to aid the second-round evaluation process. For instance, a 20-minute video presentation of the paper could be very valuable to a broader evaluating committee, or an additional two-page document highlighting the paper’s main contribution and (anticipated) impact on the field and beyond could also facilitate the assessment during this final evaluation phase.

The specifics of this second-round evaluation are yet to be determined; however, the objective is to establish a more collaborative final assessment process that minimizes the reliance on individual editors for making critical decisions. Moreover, we intend to establish clear and understandable criteria for a four and five SQARE score, perhaps defining it as a fixed fraction of top-tier submissions from each evaluation cycle, recalibrated only over extended periods.

We also want to explicitly recognize and value the significant contributions of editors at leading Economics and Econometrics journals. Our initiative does not aim to undermine their work but rather to augment it by proposing a more streamlined system for them to operate in. We hope to engage many of these same experts in SQARE, believing that their experience will greatly enhance our platform’s effectiveness.

Moving forward

We invite those who share our vision to support the SQARE initiative by signing through the form linked below, and we also invite all other comments and feedback through the corresponding feedback form. The next steps will include assembling a group of committed supporters, refining the editorial process outlined above, creating a suitable web platform to facilitate this system, and finally launching SQARE as a new quality assessment platform for Econometrics research.

Our initial approach involves launching a single evaluation system specialized in Econometrics, but we understand the value of fostering both competition and experimentation. We hope that the SQARE initiative will inspire other entities both within and beyond Econometrics to adopt, adapt, and improve upon our evaluation approach, thus cultivating a collaborative and competitive scholarly evaluation ecosystem suitable for the 21st century.

Frank DiTraglia and Martin Weidner kicked off this initiative in May 2024. We’re eager to welcome more people on board soon.

Add Your Signature, Send Us Feedback

Signatories