Random forests for regression as a weighted sum of k-Potential Nearest Neighbors

Abstract

In this paper, we tackle the problem of random forests for regression expressed as weightedsums of datapoints. We study the theoretical behavior ofk-potential nearest neighbors (k-PNNs) underbagging and obtain an upper bound on the weights of a datapoint for random forests with any type of splittingcriterion, provided that we use unpruned trees that stop growing only when there arekor less datapoints attheir leaves. Moreover, we use the previous bound together with the concept of b-terms (i.e., bootstrap terms)introduced in this paper, to derive the explicit expression of weights for datapoints in a random (k-PNNs)selection setting, a datapoint selection strategy that we also introduce and to build a framework to derive otherbagged estimators using a similar procedure. Finally, we derive from our framework the explicit expression ofweights of a regression estimate equivalent to a random forest regression estimate with the random splittingcriterion and demonstrate its equivalence both theoretically and practically.