Sample Size vs. Bias in Defect Prediction

Resumen

Most empirical disciplines promote the reuse and sharing of datasets,
as it leads to greater possibility of
replication.
While this is increasingly the case in Empirical Software Engineering,
some of the most popular bug-fix datasets are now known to be biased.
This raises two significants concerns: first, that sample bias
may lead to underperforming prediction models, and second, that
the external validity of the studies based on biased datasets may be suspect.
This issue has raised considerable consternation in the ESE literature in recent
years. However, there is a confounding factor
of these datasets that has not been examined carefully: size.
Biased datasets are sampling only some of the data
that could be sampled, and doing so in a biased fashion; but biased
samples could be smaller, or larger. Smaller data sets in general
provide less reliable bases for estimating models, and thus could
lead to inferior model performance.
In this setting, we ask the question,
what affects performance more? bias, or size?
We conduct a detailed, large-scale meta-analysis, using simulated
datasets sampled with bias from a high-quality dataset which is
relatively free of bias.
Our results suggest that size always matters just as much bias direction, and
in fact much more than bias direction
when considering information-retrieval measures such as
AUC and F-score.
This indicates that at least for prediction models, even when dealing with sampling bias,
simply finding larger samples can sometimes be sufficient. Our analysis also exposes
the complexity of the bias issue, and raises further issues to be explored in the future.