Adopting configuration management principles for managing experiment materials in families of experiments

Abstract

Context. Replication is a key component of experimentation for verifying previous results and findings. Experiment replication requires products like documentation describing the baseline experiment and a version of the experimental material. When replicating an experiment, changes may have to be made to some of the products, leading to new or modified versions of materials. After the replication has been conducted, part of or all the materials should be added to the family history or to the baseline experiment documentation. As the number of replications increases, more versions of the materials are generated. This can lead to product management chaos in replications sharing the same protocol. Objective. The aim of this paper is to adopt configuration management principles to manage experimental materials. We apply and validate these principles in a code inspection technique comparison experimentand a personality quasi-experiment. Method. The study was conducted within a research group with lengthy experience in experiment replication. This research group has had trouble with the management of the materials used to run some of the experiments replicated by other colleagues. This is a suitable context for applying action research. We used action research to adopt the configuration management principles and build a materials management framework. Result. We generated the instances of an experiment and a quasi-experiment, identifying the status and traceability of the materials. Additionally, we documented the work load required for instantiation in person-hours. We also checked the ease of use and understanding of the framework for instantiating the personality quasi-experiment configuration plan executed by researchers who did not develop the framework, as well as its usefulness for managing the experimental materials. Conclusion. The experimental materials management framework is useful for establishing the status and traceability of the experimental materials. Additionally, it improves the storage, search, location and retrieval of the experimental material versions.