The original 2024 proposal

This page is an archive. It reproduces the SQARE proposal of May 2024 and the FAQ that accompanied it, apart from this note, a heading added to the FAQ, and a few corrected typos. The two rounds of refereeing described here have since been replaced by an automated correctness check followed by comparison of papers in pairs, judged by a large editorial college, and the form of the score is being settled afresh: see the home page. The sign-up and feedback forms mentioned below are closed — write to contact@sqare.org — and the 120 people who signed are listed on the original supporters page.

Scholarly Quality Assessment for Research in Econometrics

The SQARE initiative aims to introduce a novel quality assessment system for research articles in Econometrics, offering an alternative to the conventional quality signal provided by academic journals. We envision establishing an online platform akin to a journal where authors can submit their papers for evaluation by referees and editors. However, rather than publishing the articles themselves, our focus will be purely on determining and disseminating quality assessments of each submission. This eliminates the need for a simplistic accept/reject decision on manuscripts. It also enables us to immediately compete with leading journals, because authors can submit their work to SQARE concurrently with journal submissions. Our goal is to modernize the existing academic evaluation framework by providing a more efficient, nuanced, and collaborative research quality assessment.

Background

Aspects of the following discussion may be equally relevant to other academic disciplines, both within and beyond Economics, but our focus here is on Econometrics, in line with the initial objective of the SQARE initiative.

Academic journals have been instrumental in the development and dissemination of Econometric research for many decades. Prestigious journals such as The Review of Economics and Statistics (established in 1919), Econometrica (1933), The Review of Economic Studies (1933), the Journal of Econometrics (1973), the Journal of Time Series Analysis (1980), Econometric Reviews (1982), the Journal of Business & Economic Statistics (1983), Econometric Theory (1985), the Journal of Applied Econometrics (1986), and the Econometrics Journal (1998), among others, have played pivotal roles in shaping the field. Many of our academic role models have played an integral role in founding and editing these journals, and our appreciation for the historic role of these journals and the dedicated individuals who have steered them through the years is completely separate from any debate on the continuing relevance of the existing academic journal model for the future.

The academic journal, as we recognize it today, was fully developed within the 20th century, well before the advent of the internet, email, or the World Wide Web. These journals have historically fulfilled several functions:

• Research Dissemination
• Preservation and Archiving
• Establishment of Citable References
• Quality Assurance and Evaluation through Peer Review

In the digital age of the 21st century, platforms like arXiv have increasingly assumed responsibilities for research dissemination, preservation, archiving, and establishing citable references, significantly diminishing the need for academic journals to perform these roles. Nowadays, the primary function of Econometric journals lies in quality assurance — they serve as a robust indicator of quality, highly regarded in academic assessments across various domains. However, we are concerned that this journal publication system is becoming increasingly inefficient for two main reasons:

1. While essential for conventional journals, the traditional accept/reject decision model proves increasingly inadequate for authors and reviewers alike. Within today’s highly competitive academic landscape, authors frequently face the need to sequentially submit their manuscripts to multiple journals before achieving publication. A modern quality assessment system would bypass this binary decision, offering a more nuanced and efficient evaluation approach.
2. Despite the utilization of internet platforms and email communication, the editorial and publication processes of academic journals remain largely unchanged from the days of postal correspondence. This persistence in old methodologies fails to leverage the advantages offered by modern web platforms, which could substantially improve the experience for all parties involved and introduce a more collaborative decision-making process not solely reliant on individual editors.

Although there have been attempts to move away from the binary accept/reject model (e.g., the establishment of Berkeley Electronic Press in 1999 or the practice of publishing some rejected Econometrica articles in QE), the fundamental issue (1) seems structurally ingrained within the traditional journal framework. While modernizing the editorial process (2) is theoretically feasible within existing journals, significant innovation is unlikely to stem from these institutions due to the plethora of stakeholders involved (editors, trustees, associations, publishers, etc.) that would need to agree on such changes.

If this analysis is accurate, it suggests that real change and innovation in the academic evaluation system are unlikely to come from journals, whether existing or newly created. Instead, we propose that significant advancements could arise from the establishment of quality assessment systems dedicated exclusively to quality assurance. These systems would leave the tasks of dissemination, preservation, and reference establishment to existing platforms like the Econometrics arXiv.

Several platforms are already reshaping traditional academic publishing and feedback mechanisms. For instance, ‘The Unjournal’ and the ‘Northwestern Research Feedback Project’ introduce innovative methods to enhance research through various stages of feedback and evaluation. However, these differ significantly from our proposal. Most crucially, SQARE is dedicated exclusively to assessing research quality within the field of Econometrics.

A potential challenge in transitioning from traditional academic journals to a pure quality evaluation system might be the financial model. Academic journals currently depend on publishing articles for revenue, whereas a system focused solely on quality assessment would not publish articles, affecting its revenue stream. However, the traditional journal subscription model is already waning, and though the open-access model seems more viable, open-access publication fees essentially just redirect funds that could easily support research directly. In any case, the financial aspect, while worthy of discussion, may not present a substantial obstacle to innovation, because the bulk of editorial and refereeing work is already performed voluntarily today, strongly suggesting that financial constraints will not be prohibitive to change.

The SQARE score

Given our exclusive focus on quality assessment, we can move beyond the conventional binary decision of acceptance or rejection. We propose to evaluate papers on a “five-SQARE scale” (♦ to ♦♦♦♦♦), guaranteeing that every technically correct and readable submitted paper receives a definitive assessment. This approach significantly streamlines the quality assessment process, addressing the current system’s inefficiencies where authors repeatedly submit their work to multiple journals, duplicating refereeing and editorial efforts and increasing publication lags.

In our system, a three-SQARE score (♦♦♦) aims to reflect the quality of leading field journals such as the Journal of Econometrics. A five-SQARE score (♦♦♦♦♦) is designed to correspond to top-tier general interest journals like Econometrica or The Review of Economic Studies. Assigning, communicating, and archiving such a SQARE score for every paper submitted will be the primary goal of the new evaluation system that we propose.

We anticipate that the SQARE score will serve as a vital indicator of research quality, equivalent to the role of journal publications today. This metric will be beneficial for authors themselves to gain independent feedback on how their work is received, particularly for informing decisions on their future research endeavours. Moreover, for individuals other than the authors, thoroughly reading and effectively assessing a paper can be challenging. Consequently, a dependable external quality signal becomes crucial for evaluating publication records in contexts such as hiring, promotions, grant applications, and other professional evaluations. Finally, akin to how journals circulate newsletters and social media updates about newly published or accepted papers, we can similarly disseminate updates on newly evaluated papers.

During a transition period, until the SQARE score is well-established, we anticipate that authors will continue submitting to both SQARE and traditional journals. Once SQARE earns a reliable reputation, we expect that it—and potentially other similar systems—could largely render traditional journal submissions obsolete, thereby fully realizing the efficiency gains for the academic community. The duration and extent of this transition are, of course, difficult to predict. Alternatively, should academic journals be compelled by competition to modernize and address the described inefficiencies themselves, this too could represent a successful realization of our initiative’s goals.

First-round evaluation

Upon submission to SQARE, each paper undergoes an initial evaluation that mirrors the process of one refereeing round at top field journals such as the Journal of Econometrics or the Journal of Business & Economic Statistics. This involves an editor, associate editor, and referees in their conventional roles, but with notable modifications to enhance the process.

Possible first-round outcomes:

0. Reject
1. One SQARE (♦)
2. Two SQARE (♦♦)
3. Three SQARE or higher (♦♦♦, ♦♦♦♦, ♦♦♦♦♦)

Outcomes (0), (1), and (2) conclude the evaluation process for a paper (though authors may choose to withdraw their paper before the evaluation is announced publicly). Outcome (3) offers authors a choice: accept a three SQARE score (♦♦♦) or proceed to a second-round evaluation for a potentially higher rating.

Streamlined refereeing:

There will be one single round of refereeing, and it is not designed to mandate significant revisions or request extensive rewrites from authors, except for the correction of identified mistakes or major problems. It’s important to note, however, that the clarity and presentation of a paper will naturally influence its quality score. Authors are, therefore, encouraged to submit papers that are polished and finalized. This approach is also a strategic choice, acknowledging that, at least initially, our evaluation system will operate alongside traditional journals, that is, authors are likely to submit their papers to both platforms and should not be burdened with conflicting demands for substantial revisions from multiple sources.

Innovative features:

Possible improvements to the standard evaluation process include facilitating communication between anonymous referees and authors within a single review cycle and allowing associate editors to select or indicate their preferences for which manuscript to handle. Using the full flexibility of modern internet platforms should markedly improve the editorial process efficiency.

Once a paper clears the first-round evaluation without being rejected, it is deemed technically sound and free of significant issues, concluding the responsibilities of the associate editor and referees—even if the paper receives an outcome of (3) and the authors opt for a second-round evaluation. Of course, peer review is never perfect, but a single round of refereeing, augmented by improved communication channels between authors and referees during this phase, should be sufficient to uphold the conventional standard of scrutiny.

Second-round evaluation

The first-round evaluation phase outlined above incorporates several forward-thinking elements but remains fundamentally similar to the procedures of current academic journals. We have confidence in this process’s capability to accurately assign one, two, and three SQARE scores (♦, ♦♦, ♦♦♦), based on the expertise of many competent associate editors within the field — these associate editors, with the assistance of skilled referees and under the management of a few main editors, are usually well-equipped to make these evaluations effectively.

However, distinguishing between a three, four, or five SQARE (♦♦♦, ♦♦♦♦, ♦♦♦♦♦) manuscript presents a more complex and subjective challenge, justifying a larger departure from traditional journal practices. For these higher distinctions, we propose a second-round evaluation that draws on the collective wisdom of a broader assembly, such as an editorial board or an “editorial college”, to foster a more collaborative and inclusive decision-making process. This could take the form of a simple discussion and voting mechanism, with the median vote (or some other quantile) determining the final score. Notably, the online publishing platform ‘Scipost’ already successfully employs community engagement and voting in its editorial process, underscoring the practicality and benefits of such an approach.

Should authors opt for a second-round evaluation, having received the highest possible rating in the first round, they would not only have the opportunity to submit a revised version of their paper but would also be asked to provide additional materials to aid the second-round evaluation process. For instance, a 20-minute video presentation of the paper could be very valuable to a broader evaluating committee, or an additional two-page document highlighting the paper’s main contribution and (anticipated) impact on the field and beyond could also facilitate the assessment during this final evaluation phase.

The specifics of this second-round evaluation are yet to be determined; however, the objective is to establish a more collaborative final assessment process that minimizes the reliance on individual editors for making critical decisions. Moreover, we intend to establish clear and understandable criteria for a four and five SQARE score, perhaps defining it as a fixed fraction of top-tier submissions from each evaluation cycle, recalibrated only over extended periods.

We also want to explicitly recognize and value the significant contributions of editors at leading Economics and Econometrics journals. Our initiative does not aim to undermine their work but rather to augment it by proposing a more streamlined system for them to operate in. We hope to engage many of these same experts in SQARE, believing that their experience will greatly enhance our platform’s effectiveness.

Moving forward

We invite those who share our vision to support the SQARE initiative by signing through the form linked below, and we also invite all other comments and feedback through the corresponding feedback form. The next steps will include assembling a group of committed supporters, refining the editorial process outlined above, creating a suitable web platform to facilitate this system, and finally launching SQARE as a new quality assessment platform for Econometrics research.

Our initial approach involves launching a single evaluation system specialized in Econometrics, but we understand the value of fostering both competition and experimentation. We hope that the SQARE initiative will inspire other entities both within and beyond Econometrics to adopt, adapt, and improve upon our evaluation approach, thus cultivating a collaborative and competitive scholarly evaluation ecosystem suitable for the 21st century.

Frank DiTraglia and Martin Weidner kicked off this initiative in May 2024. We’re eager to welcome more people on board soon.

Frequently asked questions (2024)

How does SQARE differ from traditional journals?

SQARE will focus solely on assessing quality: we’ll leave dissemination and publication to the arXiv. In place of the binary accept / reject decision of traditional journals, SQARE will use a two-stage evaluation process, resulting in a “SQARE score” ranging from 0 (reject) to 5 (top general interest publication). Stage 1 rates papers from 0 (reject) to 3 (top field publication) based on technical correctness, clarity of exposition, and novelty. Authors may be asked to make small corrections as part of Stage 1 but they will never be asked to substantially re-write or alter their papers. Papers awarded a 3 are eligible for a second round evaluation in which their score could increase to 4 or 5. In this stage an “editorial college” rates papers based on the importance of their contribution. Unlike at top general interest journals, there will be no “artificial scarcity” of 5 SQARE papers, and decisions will be made by a committee rather than a single editor.

Does submitting to SQARE prevent me from sending my paper to a traditional journal?

Since SQARE is not a journal, you can submit to it before, after, or at the same time as you submit to a traditional journal. Until SQARE becomes established, we expect that most people will submit to both.

Why would anyone bother to submit to SQARE?

At first, we expect that researchers will submit to SQARE for altruistic reasons: to play a role in changing econometrics for the better. The cost is low because submitting to SQARE does not preclude submitting to traditional journals, either concurrently or subsequently. There could even be reasons to submit to SQARE after being accepted at a traditional journal. If you’re coming up for tenure and don’t have time to try your luck at the top general interest journals, you might submit to a field journal for a quick acceptance. A post-publication SQARE score could show that your work is top-tier regardless of where it was published. Our aim is for SQARE to gain enough credibility to be used in tenure and promotion decisions. If that happens, SQARE’s streamlined review process will offer researchers a compelling reason to submit. When a score of 5 from SQARE counts as much as a paper in Econometrica, why spend time and effort navigating multiple revise-and-resubmit rounds at a top general interest journal?

Will SQARE scores actually matter for my career?

Building a reputation takes time, but history shows that it can be done. When Quantitative Economics (QE) was founded in 2010, its future standing within the econometrics profession was unclear. Around that time, one of us recalls a senior colleague advising that a paper in QE would likely be viewed less favorably than one in the Journal of Econometrics. Today, however, the situation is reversed: many researchers regard QE as a general interest journal ranking just below the top 5 in terms of prestige. As SQARE builds up a track record of providing accurate information about the quality of research in econometrics, we expect that it too will gain recognition and respect. As it does, we expect that SQARE will begin to play a role in tenure and promotion decisions just as traditional journals do today.

Why should we expect SQARE scores to be any better than the current system of peer review?

Economics is the academic home of mechanism design. Economists have designed sophisticated auctions for radio spectrum licenses, the residency match system, and even organ transplant exchanges. Economics also provides powerful tools for understanding problems of information aggregation. But at present we don’t apply these insights to our journal review process: the status quo is largely a product of history. Simply put, there is no reason to suppose that our current system is anywhere near the global optimum. So why not experiment with alternatives? Our thinking on the details will likely evolve over time, and we believe it will be important to formally evaluate different mechanisms for SQARE. In the meantime, here are some features that we think can improve upon the status quo:

• Faster turn-arounds: SQARE will not attempt to rewrite authors’ papers; it will simply rate them.
• Two-round evaluation: The first round is primarily about correctness, clarity, and novelty rather than importance. These characteristics are much less subjective than importance. The second round restricts attention to papers with a high first-round score and only evaluates importance.
• Editorial College: Second round evaluations draw on the collective wisdom of a wider range of experts — the Editorial College — to minimize the effect of an individual editor’s personal taste on decisions that matter for people’s careers.
• Scores that Evolve: There is no reason that a SQARE score needs to be fixed for all time. If a paper turns out to be more influential than first expected, its score could be increased.
• Referee Quality Mechanisms: Authors and editors could potentially rate referees on helpfulness. These scores could inform future assignments including, perhaps, who joins the editorial college.

What will it take to get a 5 SQARE score?

A score of 5 is intended to be roughly comparable to a top general interest journal in economics today. But unlike the “top 5”, there will be no artificial scarcity at SQARE: there are no page or issue limits because SQARE is not a journal. The second round evaluation, in which scores of 4 or 5 will be awarded, will draw on input from a wide range of experts who make up the “Editorial College”. We can envision a number of different scoring rules for combining these opinions to reach an overall score. It may be important to experiment with a number of alternatives before settling on one. Whatever the precise details of the mechanism, SQARE will be completely transparent about the way that editorial opinions are combined to produce the final score.

What if I’m unhappy with the SQARE score my paper receives?

SQARE will have an appeals process in which authors can contest their score if they feel it was awarded unfairly. We are also considering offering authors the option to have their final score hidden from view, if they so choose, in which case the official record would only state that paper X was reviewed by SQARE and nothing more.

Why bother? Most attempts to reform academia fail.

We wouldn’t be doing this unless we thought there was a chance that SQARE could succeed, but we agree that success is far from certain. If we fail, our goal is to do so in a way that creates spillover benefits, advancing future efforts to reform the publication process in economics. SQARE has already created valuable common knowledge: our signatories attest to the fact that many econometricians are unhappy with the status quo and want to do something about it. We have also had a large number of interesting and fruitful conversations with people who did not feel comfortable signing, but still agreed with many of our points. The AEA/ES/RES survey on the publication process in economics, launched one month after SQARE, likewise suggests that change is in the air. If SQARE fails because traditional journals reform themselves, we would consider that a fantastic outcome!

We plan to set up a web platform to run SQARE in the coming months and to begin experimenting with new ideas for running the evaluation process. We also plan to release reports about our progress, successes and failures in the hopes of informing other initiatives similar to our own. If others learn from our attempts and ultimately out-compete SQARE, that too would be a fantastic outcome!

How can I get involved?

The best way to get involved is by signing our supporter statement. We will soon contact our supporters to ask if they would be interested in taking a more active role in the development of SQARE. The next step will be to set up an online forum or “town hall” to solicit feedback and suggestions from supporters who are interested in helping to shape SQARE. These supporters could then help us choose a SQARE governing board to handle the day-to-day running of the enterprise. Eventually we will also need a large number of qualified referees to rate papers. If you currently write 6 referee reports for traditional journals each year, you might consider lowering that to 4 and writing two reports for SQARE.

How will SQARE be funded? What will it cost to submit?

We are currently applying for grant funding to cover the initial set-up costs of designing, building, and launching a platform on which to run SQARE. Eventually some kind of submission fee may be required, but as our costs will be lower than traditional journals this fee will likely be modest. (Referees at econometrics journals typically work for free.) We have also been considering “SQARE credits” as a way of rewarding referees with free submissions in the future.

How will SQARE avoid turning into yet another “old boy’s club”?

This is one of the most common and important questions we’ve received since launching SQARE. Economics has a reputation for hierarchy, and editorial boards at the top journals have historically been dominated by scholars at a handful of institutions. Ensuring that SQARE’s pool of reviewers and editorial college members reflects the diversity of backgrounds, opinions, and research interests of the econometrics community as a whole is crucial to making this project a success. This is a hard problem, and it’s one that we need to think more about. Clear and transparent criteria for who is chosen to join the editorial college will be crucial, as will rigorous enforcement of ethical policies for who can and cannot handle or review a submission. We look forward to discussing this further with our signatories and anyone else who has suggestions for how to tackle this pivotal issue.

Will SQARE have a replication policy?

Replicable research is crucial to scientific progress. We applaud initiatives such as the pathbreaking Journal of Applied Econometrics Data Archive, which has required all papers accepted at that journal to provide replication materials since 1995, along with more recent initiatives like those at the AEA journals. For papers with a simulation or empirical component, we can envision folding replicability into the SQARE score. In the first round evaluation, we might consider capping the score of any paper that does not provide clear replication materials at 2. Replicability could likewise play a role in the second round evaluation. Authors who provide an open-source package implementing their methods could be given a “boost” in their final evaluation. It may also be interesting to consider a mechanism for submitting replications of papers rated by SQARE, similar to the idea in the next FAQ: “How can SQARE help to find mistakes in papers?” These could be incentivized by awarding “SQARE points” or free submissions to helpful replications.

How can SQARE help to find mistakes in papers?

In addition to the standard first-round refereeing, which is analogous to a journal submission, we envision that upon submission, papers will be announced on the SQARE website, via an email list, and potentially on social media. This allows the entire community to submit any potential concerns about the papers to the editor. This feedback will not be made public, but any concerns raised will be reviewed by the editors and referees and, if necessary, brought to the authors’ attention to ask for corrections.

Additionally, one could consider a distinction between “evaluating referees”, who judge the overall quality of the paper, and “technical referees”, who purely assess technical correctness. A “technical referee” does not necessarily need to be anonymous and could therefore much more efficiently resolve technical issues by direct correspondence (including potentially a zoom call) with the authors.

How do you incentivize editors and referees to help in evaluating SQARE submissions?

Once SQARE has established a positive reputation as a reliable evaluation platform, then this should pose no problem. However, we acknowledge that during the initial phase, incentivizing the majority of the community to contribute to the SQARE project might be challenging. Nevertheless, the feedback and support signatures we have received so far clearly indicate that a significant fraction of the Econometrics community is willing to invest time and effort to drive change in our publication system and support SQARE.

How does SQARE help to address the problem of steadily increasing academic competition?

We share the concern about the adverse effects of academic competition, as highlighted in works such as Akerlof’s paper on “Sins of Omission and the Practice of Economics.” While SQARE cannot change the existence of increased competition driven by the supply and demand of academic economists, a carefully designed second-stage evaluation for SQARE submissions will aim to mitigate some negative aspects. For instance, by allocating a fixed fraction of second-stage papers to achieve a 5 SQARE score and involving a broader editorial college in the evaluation process, we ensure a more equitable and comprehensive review. Additionally, one of our primary goals is to reduce some pressure on researchers by eliminating the need to repeatedly submit to multiple journals.

If SQARE is successful, then will it become a near monopoly evaluation provider? Won’t there be an even worse concentration of power than in the current system?

This is not an immediate concern. However, in the long run, if SQARE becomes the primary evaluation platform for Econometrics articles, it could pose a problem of excessive concentration. However, one would expect a competitive ecosystem of similar evaluation platforms to emerge in such a scenario. For instance, if existing journals adopt some of SQARE’s ideas to remain competitive in a modern evaluation landscape, we would welcome this development.

Will SQARE ask/help me to rewrite my paper before assigning a quality score?

The SQARE refereeing process will focus on correcting technical problems, major presentation concerns, and minor typos. For papers that are correct and readable, little or no rewriting will be required. We believe this will be a positive feature for most submitting authors. While we recognize that some authors desire more detailed feedback to improve their papers, SQARE will not provide this type of assistance. It is conceivable to create a separate mechanism for detailed feedback, but we believe this should be functionally separate from the evaluation process itself.