The Risk of Using the Q Heterogeneity Estimator for Software Engineering Experiments

Dieste Tubio, Oscar and Fernández, Enrique and García Martínez, Ramón and Juristo Juzgado, Natalia (2011). The Risk of Using the Q Heterogeneity Estimator for Software Engineering Experiments. In: "International Symposium on Empirical Software Engineering and Measurement, ESEM 2011", 19/09/2011 - 23/09/2011, Banff, Albert, Canada. ISBN 978-1-4577-2203-5. pp. 68-76.

Description

Title: The Risk of Using the Q Heterogeneity Estimator for Software Engineering Experiments
Author/s:
  • Dieste Tubio, Oscar
  • Fernández, Enrique
  • García Martínez, Ramón
  • Juristo Juzgado, Natalia
Item Type: Presentation at Congress or Conference (Article)
Event Title: International Symposium on Empirical Software Engineering and Measurement, ESEM 2011
Event Dates: 19/09/2011 - 23/09/2011
Event Location: Banff, Albert, Canada
Title of Book: Proceedings of International Symposium on Empirical Software Engineering and Measurement, ESEM 2011
Date: 2011
ISBN: 978-1-4577-2203-5
Subjects:
Freetext Keywords: Meta-analysis , effect size , heterogeneity , reliability , statistical power , weighted mean difference (WMD)
Faculty: Facultad de Informática (UPM)
Department: Lenguajes y Sistemas Informáticos e Ingeniería del Software
Creative Commons Licenses: Recognition - No derivative works - Non commercial

Full text

[img]
Preview
PDF - Requires a PDF viewer, such as GSview, Xpdf or Adobe Acrobat Reader
Download (604kB) | Preview

Abstract

All meta-analyses should include a heterogeneity analysis. Even so, it is not easy to decide whether a set of studies are homogeneous or heterogeneous because of the low statistical power of the statistics used (usually the Q test). Objective: Determine a set of rules enabling SE researchers to find out, based on the characteristics of the experiments to be aggregated, whether or not it is feasible to accurately detect heterogeneity. Method: Evaluate the statistical power of heterogeneity detection methods using a Monte Carlo simulation process. Results: The Q test is not powerful when the meta-analysis contains up to a total of about 200 experimental subjects and the effect size difference is less than 1. Conclusions: The Q test cannot be used as a decision-making criterion for meta-analysis in small sample settings like SE. Random effects models should be used instead of fixed effects models. Caution should be exercised when applying Q test-mediated decomposition into subgroups.

More information

Item ID: 11569
DC Identifier: http://oa.upm.es/11569/
OAI Identifier: oai:oa.upm.es:11569
Official URL: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6092555&contentType=Conference+Publications&sortType%3Dasc_p_Sequence%26filter%3DAND%28p_IS_Number%3A6092538%29
Deposited by: Memoria Investigacion
Deposited on: 19 Jul 2012 10:53
Last Modified: 20 Apr 2016 19:36
  • Logo InvestigaM (UPM)
  • Logo GEOUP4
  • Logo Open Access
  • Open Access
  • Logo Sherpa/Romeo
    Check whether the anglo-saxon journal in which you have published an article allows you to also publish it under open access.
  • Logo Dulcinea
    Check whether the spanish journal in which you have published an article allows you to also publish it under open access.
  • Logo de Recolecta
  • Logo del Observatorio I+D+i UPM
  • Logo de OpenCourseWare UPM