FAQs

How does SQARE differ from traditional journals?

SQARE will focus solely on assessing quality: we’ll leave dissemination and publication to the arXiv. In place of the binary accept / reject decision of traditional journals, SQARE will use a two-stage evaluation process, resulting in a “SQARE score” ranging from 0 (reject) to 5 (top general interest publication). Stage 1 rates papers from 0 (reject) to 3 (top field publication) based on technical correctness, clarity of exposition, and novelty. Authors may be asked to make small corrections as part of Stage 1 but they will never be asked to substantially re-write or alter their papers. Papers awarded a 3 are eligible for a second round evaluation in which their score could increase to 4 or 5. In this stage an “editorial college” rates papers based on the importance of their contribution. Unlike at top general interest journals, there will be no “artificial scarcity” of 5 SQARE papers, and decisions will be made by a committee rather than a single editor.

Does submitting to SQARE prevent me from sending my paper to a traditional journal?

Since SQARE is not a journal, you can submit to it before, after, or at the same time as you submit to a traditional journal. Until SQARE becomes established, we expect that most people will submit to both.

Why would anyone bother to submit to SQARE?

At first, we expect that researchers will submit to SQARE for altruistic reasons: to play a role in changing econometrics for the better. The cost is low because submitting to SQARE does not preclude submitting to traditional journals, either concurrently or subsequently. There could even be reasons to submit to SQARE after being accepted at a traditional journal. If you’re coming up for tenure and don’t have time to try your luck at the top general interest journals, you might submit to a field journal for a quick acceptance. A post-publication SQARE score could show that your work is top-tier regardless of where it was published. Our aim is for SQARE to gain enough credibility to be used in tenure and promotion decisions. If that happens, SQARE’s streamlined review process will offer researchers a compelling reason to submit. When a score of 5 from SQARE counts as much as a paper in Econometrica, why spend time and effort navigating multiple revise-and-resubmit rounds at a top general interest journal?

Will SQARE scores actually matter for my career?

Building a reputation takes time, but history shows that it can be done. When Quantitative Economics (QE) was founded in 2010, its future standing within the econometrics profession was unclear. Around that time, one of us recalls a senior colleague advising that a paper in QE would likely be viewed less favorably than one in the Journal of Econometrics. Today, however, the situation is reversed: many researchers regard QE as a general interest journal ranking just below the top 5 in terms of prestige. As SQARE builds up a track record of providing accurate information about the quality of research in econometrics, we expect that it too will gain recognition and respect. As it does, we expect that SQARE will begin to play a role in tenure and promotion decisions just as traditional journals do today.

Why should we expect SQARE scores to be any better than the current system of peer review?

Economics is the academic home of mechanism design. Economists have designed sophisticated auctions for radio spectrum licenses, the residency match system, and even organ transplant exchanges. Economics also provides powerful tools for understanding problems of information aggregation. But at present we don’t apply these insights to our journal review process: the status quo is largely a product of history. Simply put, there is no reason to suppose that our current system is anywhere near the global optimum. So why not experiment with alternatives? Our thinking on the details will likely evolve over time, and we believe it will be important to formally evaluate different mechanisms for SQARE. In the meantime, here are some features that we think can improve upon the status quo:

• Faster turn-arounds: SQARE will not attempt to rewrite authors' papers; it will simply rate them.
• Two-round evaluation: The first round is primarily about correctness, clarity, and novelty rather than importance. These characteristics are much less subjective than importance. The second round restricts attention to papers with a high first-round score and only evaluates importance.
• Editorial College: Second round evaluations draw on the collective wisdom of a wider range of experts–the Editorial College–to minimize the effect of an individual editor’s personal taste on decisions that matter for peoples' careers.
• Scores that Evolve: There is no reason that a SQARE score needs to be fixed for all time. If a paper turns out to be more influential than first expected, its score could be increased.
• Referee Quality Mechanisms: Authors and editors could potentially rate referees on helpfulness. These scores could inform future assignments including, perhaps, who joins the editorial college.

What will it take to get a 5 SQARE score?

A score of 5 is intended to be roughly comparable to a top general interest journal in economics today. But unlike the “top 5”, there will be no artificial scarcity at SQARE: there are no page or issue limits because SQARE is not a journal. The second round evaluation, in which scores of 4 or 5 will be awarded, will draw on input from a wide range of experts who make up the “Editorial College''. We can envision a number of different scoring rules for combining these opinions to reach an overall score. It may be important to experiment with a number of alternatives before settling on one. Whatever the precise details of the mechanism, SQARE will be completely transparent about the way that editorial opinions are combined to produce the final score.

What if I’m unhappy with the SQARE score my paper receives?

SQARE will have an appeals process in which authors can contest their score if they feel it was awarded unfairly. We are also considering offering authors the option to have their final score hidden from view, if they so choose, in which case the official record would only state that paper X was reviewed by SQARE and nothing more.

Why bother? Most attempts to reform academia fail.

We wouldn’t be doing this unless we thought there was a chance that SQARE could succeed, but we agree that success is far from certain. If we fail, our goal is to do so in a way that creates spillover benefits, advancing future efforts to reform the publication process in economics. SQARE has already created valuable common knowledge: our signatories attest to the fact that many econometricians are unhappy with the status quo and want to do something about it. We have also had a large number of interesting and fruitful conversations with people who did not feel comfortable signing, but still agreed with many of our points. The AEA/ES/RES survey on the publication process in economics, launched one month after SQARE, likewise suggests that change is in the air. If SQARE fails because traditional journals reform themselves, we would consider that a fantastic outcome!

We plan to set up a web platform to run SQARE in the coming months and to begin experimenting with new ideas for running the evaluation process. We also plan to release reports about our progress, successes and failures in the hopes of informing other initiatives similar to our own. If others learn from our attempts and ultimately out-compete SQARE, that too would be a fantastic outcome!

How can I get involved?

The best way to get involved is by signing our supporter statement. We will soon contact our supporters to ask if they would be interested in taking a more active role in the development of SQARE. The next step will be to set up an online forum or “town hall” to solicit feedback and suggestions from supporters who are interested in helping to shape SQARE. These supporters could then help us choose a SQARE governing board to handle the day-to-day running of the enterprise. Eventually we will also need a large number of qualified referees to rate papers. If you currently write 6 referee reports for traditional journals each year, you might consider lowering that 4 and writing two reports for SQARE.

How will SQARE be funded? What will it cost to submit?

We are currently applying for grant funding to cover the initial set-up costs of designing, building, and launching a platform on which to run SQARE. Eventually some kind of submission fee may be required, but as our costs will be lower than traditional journals this fee will likely be modest. (Referees at econometrics journals typically work for free.) We have also been considering “SQARE credits” as a way of rewarding referees with free submissions in the future.

How will SQARE avoid turning into yet another “old boy’s club”?

This is one of the most common and important questions we’ve received since launching SQARE. Economics has a reputation for hierarchy, and editorial boards at the top journals have historically been dominated by scholars at a handful of institutions. Ensuring that SQARE’s pool of reviewers and editorial college members reflects the diversity of backgrounds, opinions, and research interests of the econometrics community as a whole is crucial to making this project a success. This is a hard problem, and it’s one that we need to think more about. Clear and transparent criteria for who is chosen to join the editorial college will be crucial, as will rigorous enforcement of ethical policies for who can and cannot handle or review a submission. We look forward to discussing this further with our signatories and anyone else who has suggestions for how to tackle this pivotal issue.

Will SQARE have a replication policy?

Replicable research is crucial to scientific progress. We applaud initiatives such as the pathbreaking Journal of Applied Econometrics Data Archive, which has required all papers accepted at that journal to provide replication materials since 1995, along with more recent initiatives like those at the AEA journals. For papers with a simulation or empirical component, we can envision folding replicability into the SQARE score. In the first round evaluation, we might consider capping the score of any paper that does not provide clear replication materials at 2. Replicability could likewise play a role in the second round evaluation. Authors who provide an open-source package implementing their methods could be given a “boost” in their final evaluation. It may also be interesting to consider a mechanism for submitting replications of papers rated by SQARE, similar to the idea in the next FAQ: “How can SQARE help to find mistakes in papers?” These could be incentivized by awarding “SQARE points” or free submissions to helpful replications.

How can SQARE help to find mistakes in papers?

In addition to the standard first-round refereeing, which is analogous to a journal submission, we envision that upon submission, papers will be announced on the SQARE website, via an email list, and potentially on social media. This allows the entire community to submit any potential concerns about the papers to the editor. This feedback will not be made public, but any concerns raised will be reviewed by the editors and referees and, if necessary, brought to the authors' attention to ask for corrections.

Additionally, one could consider a distinction between “evaluating referees”, who judge the overall quality of the paper, and “technical referees”, who purely assess technical correctness. A “technical referee” does not necessarily need to be anonymous and could therefore much more efficiently resolve technical issues by direct correspondence (including potentially a zoom call) with the authors.

How do you incentivize editors and referees to help in evaluating SQARE submissions?

Once SQARE has established a positive reputation as a reliable evaluation platform, then this should pose no problem. However, we acknowledge that during the initial phase, incentivizing the majority of the community to contribute to the SQARE project might be challenging. Nevertheless, the feedback and support signatures we have received so far clearly indicate that a significant fraction of the Econometrics community is willing to invest time and effort to drive change in our publication system and support SQARE.

How does SQARE help to address the problem of steadily increasing academic competition?

We share the concern about the adverse effects of academic competition, as highlighted in works such as Akerlof’s paper on “Sins of Omission and the Practice of Economics.” While SQARE cannot change the existence of increased competition driven by the supply and demand of academic economists, a carefully designed second-stage evaluation for SQARE submissions will aim to mitigate some negative aspects. For instance, by allocating a fixed fraction of second-stage papers to achieve a 5 SQARE score and involving a broader editorial college in the evaluation process, we ensure a more equitable and comprehensive review. Additionally, one of our primary goals is to reduce some pressure on researchers by eliminating the need to repeatedly submit to multiple journals.

If SQARE is successful, then will it become a near monopoly evaluation provider? Won’t there be an even worse concentration of power than in the current system?

This is not an immediate concern. However, in the long run, if SQARE becomes the primary evaluation platform for Econometrics articles, it could pose a problem of excessive concentration. However, one would expect a competitive ecosystem of similar evaluation platforms to emerge in such a scenario. For instance, if existing journals adopt some of SQARE’s ideas to remain competitive in a modern evaluation landscape, we would welcome this development.

Will SQARE ask/help me to rewrite my paper before assigning a quality score?

The SQARE refereeing process will focus on correcting technical problems, major presentation concerns, and minor typos. For papers that are correct and readable, little or no rewriting will be required. We believe this will be a positive feature for most submitting authors. While we recognize that some authors desire more detailed feedback to improve their papers, SQARE will not provide this type of assistance. It is conceivable to create a separate mechanism for detailed feedback, but we believe this should be functionally separate from the evaluation process itself.