On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices

While learning with limited labelled data can effectively deal with a lack of
labels, it is also sensitive to the effects of uncontrolled randomness introduced
by so-called randomness factors (i.e., non-deterministic decisions such as choice
or order of samples). We propose and formalise a method to systematically
investigate the effects of individual randomness factors while taking the
interactions (dependence) between them into consideration. To this end, our
method mitigates the effects of other factors while observing how the performance
varies across multiple runs. Applying our method to multiple randomness factors
across in-context learning and fine-tuning approaches on 7 representative text
classification tasks and meta-learning on 3 tasks, we show that: 1) disregarding
interactions between randomness factors in existing works led to inconsistent
findings due to incorrect attribution of the effects of randomness factors, such
as disproving the consistent sensitivity of in-context learning to sample order
even with random sample selection; and 2) besides mutual interactions, the
effects of randomness factors, especially sample order, are also dependent on
more systematic choices unexplored in existing works, such as number of classes,
samples per class or choice of prompt format.