Metabias

An algorithm needs some method of altering its base bias as it is presented with sets of instances. The major claim of this paper is that, as the vast majority of algorithms are intended for use on a collection of problem instances, that meta bias is therefore necessary. Evidence for the fact that algorithms are intended to be reused is that the proponents of newly proposed algorithm test them on a number of problem instances which are reported in their papers. Meta bias is any mechanism which can adjust the base bias (see figure 2 i.e. it is a probability distribution over a set of base biases).

One of the most important metabiases is reporting bias. Only about 50% of randomised trials ultimately reach publication in a journal indexed in a major electronic database

Given the topic of this review, formal assessment of metabiases such as publication bias or selective reporting within studies was not feasible.

Heterogeneity across the studies was tested with the Cochran Q and I2 statistics. We considered an I2 statistic of >25% as a low degree of heterogeneity, >50% as moderate heterogeneity, and >75% as high heterogeneity. To ascertain potential meta-bias, a funnel plot was qualitatively assessed for asymmetry.

Metabiases / We will plot study sample size against effect size to evaluate whether these are skewed and asymmetrical in the presence of publication bias and other biases. In addition, the risk of bias assessment tools will identify whether studies indicate selective reporting of outcomes, which we will summarise across studies if possible.