Williams, Christopher

Hennig, Matthias

Eastwood, Cian

School of Informatics at The University of Edinburgh

2024-03-18T13:06:09Z

2024-03-18T13:06:09Z

2024-03-18

Machine learning systems have made headlines in recent years, defeating world champions in Go, enhancing medical diagnoses, and redefining how we work with tools like ChatGPT. However, despite these impressive feats, machine learning systems remain fragile when faced with test data that differs from their training data. This fragility stems from a fundamental mismatch between textbook machine-learning methods and their real-world application. While textbook methods assume that the conditions under which a system is developed are similar to those in which it is deployed, in reality, systems tend to be developed under one set of conditions (e.g., in a lab) and deployed to another (e.g., a clinic). As a result, many machine learning systems are not prepared for the condition differences or distribution shifts they face upon deployment, leading to some high-profile and costly failures. For safety-critical settings like healthcare and autonomous driving, such failures represent a major barrier to real-world deployment. In this thesis, I argue that we must first accept that shift happens, and subsequently focus on how we can best prepare. To do so, I present four of my works that illustrate how machine learning systems can be prepared for (and adapted to) real-world distribution shifts. Together, these contributions take us closer to reliable machine learning systems that can be deployed in safety-critical settings. In the first work, the setting is source-free domain adaptation, i.e., adapting a model to unlabelled test data without the original training data. Here, we prepare for a change in measurement device (e.g., X-rays from a different scanner) by storing lightweight statistics of the training data. By restoring these statistics on the test data, we see improved accuracy, calibration and data efficiency over prior methods. In the second work, the setting is domain generalisation, i.e., performing well on test data from new environments or domains by leveraging data from multiple related domains at training time. Here, we prepare for more flexible and unknown changes by exploiting invariances across the training domains that hold with high probability in unseen test domains. In particular, by minimising a particular quantile of a model's performance distribution over domains, we learn models that perform well with the corresponding probability. In the third work, the setting is again domain generalisation, but this time we focus on ways to harness so-called "spurious'' features without test-domain labels. In particular, we show that predictions based on invariant/stable features can be used to adapt our usage of spurious/unstable features to new test domains, so long as the stable and unstable features are complementary (i.e., conditionally independent given the label). By safely harnessing complementary spurious features, we boost performance without sacrificing robustness. Finally, in the fourth work, the setting is disentangled representation learning which, in the context of this thesis, can be viewed as preparing for a change in the task itself by recovering and separating the underlying factors of variation. To this end, we extend an existing evaluation framework by first introducing a measure of representation explicitness or ease of use, and then connecting the framework to identifiability.

https://hdl.handle.net/1842/41628

http://dx.doi.org/10.7488/era/4359

en

The University of Edinburgh

Eastwood, C., Mason, I., Williams, C. K. I., and Schölkopf, B. (2022a). Sourcefree adaptation to measurement shift via bottom-up feature restoration. In The Tenth International Conference on Learning Representations

Eastwood, C., Robey, A., Singh, S., Kügelgen, J. V., Hassani, H., Pappas, G. J., and Schölkopf, B. (2022c). Probable domain generalization via quantile risk minimization. In Advances in Neural Information Processing Systems, volume 35, pages 17340–17358

Eastwood, C., Singh, S., Nicolicioiu, L. A., Von Kügelgen, J., and Schölkopf, B. (2023b). Spuriosity didn’t kill the classifier: Using invariant predictions to harness spurious features. In Advances in Neural Information Processing Systems

Eastwood, C., Nicolicioiu, A. L., Kügelgen, J. V., Keki´c, A., Träuble, F., Dittadi, A., and Schölkopf, B. (2023a). DCI-ES: An extended disentanglement framework with connections to identifiability. In The Eleventh International Conference on Learning Representations

machine learning systems

real-world distribution shifts

source-free domain adaptation

unlabelled test data without the original training data

Shift happens: how can machine learning systems be best prepared?

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy