Why should this paper be published?

Here is a simple suggestion that will immensely improve the publishing/reviewing experience.

Every submission should include a subsection called: "Why should this paper be published?" (WSTPBP).

Here, the authors should provide an objective/falsifiable statement of why this paper deserves publication. This may include statements like:

  • this feature of the system is novel (no previous work has it)
  • this is a more {fault-tolerant, secure, efficient, simple} solution than previous solutions

This should be a targeted minimal set. If one of these statements is falsified by the reviewers, it could be a valid reason to reject the paper.

In that sense, WSTPBP is the authors' contract with the reviewers. The authors put a stake in the ground, saying that the reviewers should focus on the claims in WSTPBP, rather than giving other pretexts, diverging to unclaimed features to reject the paper.

This focuses the reviewing process as the reviewers evaluate the paper based on WSTPBP. If they can falsify one of the claims, they can call for a reject. The cause of rejection should be given as this and that item being falsified due to such and such reasons.

It is possible that some authors may provide a low-bar/low-standards WSTPBP section to game this system. The reviewers can still prescribe a reject, if the majority of the reviewers rule that the WSTPBP as provided is insufficient to merit a publication. In this case, the program committee (PC) chair and other PC members can be involved as an additional control mechanism. It is relatively easy to evaluate the adequacy of WSTPBP, because the contract is about what, rather than how.

Benefits of WSTPBP

WSTPBP decomposes the problem into two subproblems:

  • Is the WSTPBP legitimate?  
  • Does the paper fulfil the claims in WSTPBP?

WSTPBP helps focus the authors' claims and make them explicit. Today, most submissions do not answer the "Why should this paper be published?" question. There is a contributions subsection in most papers, but most papers use this as a fluffy way of pumping up the paper. They say things like we evaluated against this and that, we investigated this and that. They talk about how, rather than what. They fail to make objective/falsifiable claims about why the paper deserves publication. WSTPBP serves as a specification for the system/protocol/solution proposed in the paper. You can't believe how many papers skip this. This problem got much worse in the last decade or so.

WSTPBP keeps the reviewers accountable as well. It makes it harder for the reviewers to make up pretexts to reject the paper as it forces the reviewers to evaluate the paper with respect to its contract. If the reviewers cannot falsify the contract, but argue that the contract is insufficient, this requires the PC chair should double-check the contract.

WSTPBP also improves the rebuttal and resubmission process by focusing the discussion.

There is of course no silver bullet. There may be uncertainties in the degrees of satisfying a contract. A reviewer may say that the paper does not provide enough evidence to justify a certain claim, and the authors may disagree with this claim. However, the act of writing WSTPBP and evaluating with respect to it reduces the target size and improves the process significantly.

Comments

Anonymous said…
Having a WSTPBP section in general is good practice, although I am a bit curious how different this is from the abstract. I think the abstract section already serves the "why should I read this paper" function, for readers at least. Perhaps the reviewing process should just make sure the abstract section is well written for the intended purposes.

Popular posts from this blog

The end of a myth: Distributed transactions can scale

Hints for Distributed Systems Design

Foundational distributed systems papers

Learning about distributed systems: where to start?

Metastable failures in the wild

Scalable OLTP in the Cloud: What’s the BIG DEAL?

The demise of coding is greatly exaggerated

SIGMOD panel: Future of Database System Architectures

Dude, where's my Emacs?

There is plenty of room at the bottom