Storkey, Amos

Gutmann, Michael

Toner, William

2025-05-01T09:03:02Z

2025-05-01T09:03:02Z

2025-05-01

Over the last decade, there has been a significant improvement in machine learning methods for classification, particularly in computer vision. This progress has increased the demand for large labelled datasets. However, obtaining clean, accurately labelled datasets on the required scale can be prohibitively expensive. Consequently, practitioners often resort to methods that yield larger datasets but with substantial label noise, such as web querying or crowd-sourcing. Even conventional data collection methods can introduce errors due to human fallibility, particularly in complex domains like medical imaging. While broadly beneficial, the high expressibility of neural network classifiers renders them prone to overfitting on noisy labels. This issue can severely impact model performance and generalisation, leading to significant setbacks in practical applications. The challenge of noisy labels has sparked substantial interest in developing methodologies robust to such conditions. Among various strategies for handling noisy labels, ‘robust loss functions’ have emerged as a favoured method due to their simplicity and effectiveness. Despite extensive research, important theoretical gaps in understanding robust loss functions persist. Moreover, a disconnect between the theory and practice of these functions remains. As a result, there is an ongoing lack of principled yet simple and computationally economical loss-based approaches for learning in the presence of noisy labels. This thesis addresses these challenges. We show how overfitting can be curbed by lower-bounding the training loss, motivating this loss-bounding policy theoretically. We derive the relevant lower bound, showing how it can be estimated via the label noise rate. We also develop a straightforward early-stopping policy that operates without knowledge of the noise rate or access to a cleanly-labelled validation set. We validate the effectiveness of our approaches through extensive experiments across various noisily-labelled benchmark datasets. We build on and generalise the existing theory of noise-tolerant loss functions, demonstrating that such loss functions are rare and do not exist for most noise models. This work provides important theoretical insights and establishes conditions under which a loss function can be considered noise-tolerant. Our final chapter describes how GANs can be conceptualised in terms of noisy labels, deriving a novel loss function and providing both theoretical insights and experimental results.

https://hdl.handle.net/1842/43397

http://dx.doi.org/10.7488/era/5933

en

The University of Edinburgh

Robust loss functions

Noisy labels

Overfitting

Label noise rate

Noise-tolerant

Robust loss functions for machine learning in the presence of noisy labels

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy