Quantitative Determination of the Relationship between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews

Dieste Tubio, Oscar and Juristo Juzgado, Natalia and Grimán, Anna and Saxena, Himanshu (2011). Quantitative Determination of the Relationship between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews. In: "Fifth International Symposium on Empirical Software Engineering and Measurement, ESEM 2011", 19/09/2011 - 23/09/2011, Banff, Albert, Canada. ISBN 978-1-4577-2203-5. pp. 285-294.

Description

Title: Quantitative Determination of the Relationship between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews
Author/s:
  • Dieste Tubio, Oscar
  • Juristo Juzgado, Natalia
  • Grimán, Anna
  • Saxena, Himanshu
Item Type: Presentation at Congress or Conference (Article)
Event Title: Fifth International Symposium on Empirical Software Engineering and Measurement, ESEM 2011
Event Dates: 19/09/2011 - 23/09/2011
Event Location: Banff, Albert, Canada
Title of Book: Proceedings of the Fifth International Symposium on Empirical Software Engineering and Measurement, ESEM 2011
Date: 2011
ISBN: 978-1-4577-2203-5
Subjects:
Freetext Keywords: Systematic Literature Review (SLR); Quality Assessment (QA) of experiments; Checklist; Quality Scale
Faculty: Facultad de Informática (UPM)
Department: Lenguajes y Sistemas Informáticos e Ingeniería del Software
Creative Commons Licenses: Recognition - No derivative works - Non commercial

Full text

[img]
Preview
PDF - Requires a PDF viewer, such as GSview, Xpdf or Adobe Acrobat Reader
Download (253kB) | Preview

Abstract

Quality assessment is one of the activities performed as part of systematic literature reviews. It is commonly accepted that a good quality experiment is bias free. Bias is considered to be related to internal validity (e.g., how adequately the experiment is planned, executed and analysed). Quality assessment is usually conducted using checklists and quality scales. It has not yet been proven;however, that quality is related to experimental bias. Aim: Identify whether there is a relationship between internal validity and bias in software engineering experiments. Method: We built a quality scale to determine the quality of the studies, which we applied to 28 experiments included in two systematic literature reviews. We proposed an objective indicator of experimental bias, which we applied to the same 28 experiments. Finally, we analysed the correlations between the quality scores and the proposed measure of bias. Results: We failed to find a relationship between the global quality score (resulting from the quality scale) and bias; however, we did identify interesting correlations between bias and some particular aspects of internal validity measured by the instrument. Conclusions: There is an empirically provable relationship between internal validity and bias. It is feasible to apply quality assessment in systematic literature reviews, subject to limits on the internal validity aspects for consideration.

More information

Item ID: 11571
DC Identifier: http://oa.upm.es/11571/
OAI Identifier: oai:oa.upm.es:11571
Official URL: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6092577
Deposited by: Memoria Investigacion
Deposited on: 19 Jul 2012 09:23
Last Modified: 20 Apr 2016 19:36
  • Logo InvestigaM (UPM)
  • Logo GEOUP4
  • Logo Open Access
  • Open Access
  • Logo Sherpa/Romeo
    Check whether the anglo-saxon journal in which you have published an article allows you to also publish it under open access.
  • Logo Dulcinea
    Check whether the spanish journal in which you have published an article allows you to also publish it under open access.
  • Logo de Recolecta
  • Logo del Observatorio I+D+i UPM
  • Logo de OpenCourseWare UPM