Hospedales, Timothy

Bilen, Hakan

Storkey, Amos

Bohdal, Ondrej

Engineering and Physical Sciences Research Council (EPSRC)

University of Edinburgh

Alan Turing Institute as part of the Enrichment scheme

Samsung AI Center

Amazon Web Services

2024-02-16T12:15:56Z

2024-02-16T12:15:56Z

2024-02-16

Meta-learning in the broader context concerns how an agent learns about their own learning, allowing them to improve their learning process. Learning how to learn is not only beneficial for humans, but it has also shown vast benefits for improving how machines learn. In the context of machine learning, meta-learning enables models to improve their learning process by selecting suitable meta-parameters that influence the learning. For deep learning specifically, the meta-parameters typically describe details of the training of the model but can also include description of the model itself - the architecture. Meta-learning is usually done with specific goals in mind, for example trying to improve ability to generalize or learn new concepts from only a few examples. Meta-learning can be powerful, but it comes with a key downside: it is often computationally costly. If the costs would be alleviated, meta-learning could be more accessible to developers of new artificial intelligence models, allowing them to achieve greater goals or save resources. As a result, one key focus of our research is on significantly improving the efficiency of meta-learning. We develop two approaches: EvoGrad and PASHA, both of which significantly improve meta-learning efficiency in two common scenarios. EvoGrad allows us to efficiently optimize the value of a large number of differentiable meta-parameters, while PASHA enables us to efficiently optimize any type of meta-parameters but fewer in number. Meta-learning is a tool that can be applied to solve various problems. Most commonly it is applied for learning new concepts from only a small number of examples (few-shot learning), but other applications exist too. To showcase the practical impact that meta-learning can make in the context of neural networks, we use meta-learning as a novel solution for two selected problems: more accurate uncertainty quantification (calibration) and general-purpose few-shot learning. Both are practically important problems and using meta-learning approaches we can obtain better solutions than the ones obtained using existing approaches. Calibration is important for safety-critical applications of neural networks, while general-purpose few-shot learning tests model's ability to generalize few-shot learning abilities across diverse tasks such as recognition, segmentation and keypoint estimation. More efficient algorithms as well as novel applications enable the field of meta-learning to make more significant impact on the broader area of deep learning and potentially solve problems that were too challenging before. Ultimately both of them allow us to better utilize the opportunities that artificial intelligence presents.

https://hdl.handle.net/1842/41452

http://dx.doi.org/10.7488/era/4184

en

The University of Edinburgh

Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error Bohdal, O., Yang, Y. & Hospedales, T. M., 23 Aug 2023, In: Transactions on Machine Learning Research. p. 1-21 21 p.

PASHA: Efficient HPO and NAS with Progressive Resource Allocation Bohdal, O., Balles, L., Wistuba, M., Ermis, B., Archambeau, C. & Zappella, G., 1 May 2023, The Eleventh International Conference on Learning Representations: ICLR 2023. p. 1-20

EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization Bohdal, O., Yang, Y. & Hospedales, T. M., 6 Dec 2021, 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Neural Information Processing Systems, 12 p. (Advances in Neural Information Processing Systems).

Meta Omnium: A Benchmark for General-Purpose Learning-to-Learn Bohdal, O., Tian, Y., Zong, Y., Chavhan, R., Li, D., Gouk, H., Guo, L. & Hospedales, T., 22 Aug 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, p. 7693-7703 11 p. (Conference on Computer Vision and Pattern Recognition (CVPR)).

Bohdal, O., Li, D., and Hospedales, T. (2022). Feed-forward source-free domain adaptation via class prototypes. In ECCV OOD-CV Workshop.

Bohdal, O., Li, D., Hu, S. X., and Hospedales, T. (2024). Feed-forward latent domain adaptation. In WACV.

Bohdal, O., Yang, Y., and Hospedales, T. (2020). Flexible dataset distillation: learn labels instead of images. In NeurIPS Workshop on Meta-Learning.

Dutt, R., Bohdal, O., Tsaftaris, S. A., and Hospedales, T. (2024). Fairtune: Optimizing parameter efficient fine tuning for fairness in medical image analysis. In ICLR.

Li, R., Bohdal, O., Mishra, R. K., Kim, H., Li, D., Lane, N. D., and Hospedales, T. M. (2021a). A channel coding benchmark for meta-learning. In NeurIPS Datasets and Benchmarks.

meta-learning

deep learning

machine learning

computer vision

few-shot learning

hyperparameter optimization

optimization

Meta-learning algorithms and applications

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy