The Truth, the Whole Truth, and Nothing but the Truth: A Pragmatic Guide to Assessing Empirical Evaluations

Blackburn, Stephen M, Diwan, Amer, Hauswirth, Mattias, Sweeney, Peter F, Amaral, Jose Nelson, Brecht, Tim, Bulej, Lubomr, Click, Cliff, Eeckhout, Lieven, Fischmeister, Sebastian, and others. (2016) The Truth, the Whole Truth, and Nothing but the Truth: A Pragmatic Guide to Assessing Empirical Evaluations. Transactions on Programming Languages and Systems, 38 (4). ISSN 0164-0925. E-ISSN 1558-4593. (doi:10.1145/2983574) (KAR id:55171)

PDF Author's Accepted Manuscript Language: English
Download this file (PDF/1MB)
Request a format suitable for use with assistive technology e.g. a screenreader
Official URL: http://dx.doi.org/10.1145/2983574

Abstract

An unsound claim can misdirect a field, encouraging the pursuit of unworthy ideas and the abandonment of promising ideas. An inadequate description of a claim can make it difficult to reason about the claim, for example to determine whether the claim is sound. Many practitioners will acknowledge the threat of un- sound claims or inadequate descriptions of claims to their field. We believe that this situation is exacerbated and even encouraged by the lack of a systematic approach to exploring, exposing, and addressing the source of unsound claims and poor exposition.

This paper proposes a framework that identifies three sins of reasoning that lead to unsound claims and two sins of exposition that lead to poorly described claims. Sins of exposition obfuscate the objective of determining whether or not a claim is sound, while sins of reasoning lead directly to unsound claims.

Our framework provides practitioners with a principled way of critiquing the integrity of their own work and the work of others. We hope that this will help individuals conduct better science and encourage a cultural shift in our research community to identify and promulgate sound claims.

Item Type:	Article
DOI/Identification number:	10.1145/2983574
Uncontrolled keywords:	Performance evaluation (efficiency and effectiveness); experimental evaluation; observation study; experimentation
Subjects:	Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming,
Divisions:	Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Depositing User:	Richard Jones
Date Deposited:	28 Apr 2016 12:39 UTC
Last Modified:	05 Nov 2024 10:44 UTC
Resource URI:	https://kar.kent.ac.uk/id/eprint/55171 (The current URI for this page, for reference purposes)

University of Kent Author Information

Jones, Richard E..

Creator's ORCID:	https://orcid.org/0000-0002-8159-0297
CReDIT Contributor Roles:

Kalibera, Tomas.

Creator's ORCID:
CReDIT Contributor Roles:

Depositors only (login required):

Altmetric

Download Statistics

Total unique views for this document in KAR since July 2020. For more details click on the image.