Fine-tuning foundation models for objective video quality assessment (Bachelor)

Objective video quality assessment methods aim to develop an automatic framework to predict the perceived quality of videos. To this end, state-of-the-art research draws heavy inspiration from the human visual system, to understand how sensitive we are to different aberrations that may corrupt a visual signal in a digital system.

Foundation models offer a highly generalizable backbone thanks to large-scale training and have become a quintessential part of today’s state-of-the-art. Vision foundation models feature highly accurate scene understanding, which facilitates fast adaptation to many downstream tasks, including quality assessment. However, there is more to visual quality assessment than simply understanding a scene. An accurate model must be able to capture signal distortions as well as predict how the distortions may affect a viewer’s quality of experience.

The scope of this thesis includes:

Understanding transformer-based vision foundation models
Fine-tuning such models to video quality assessment
Comparing their performance with the state-to-the-art
Analyzing their strengths and weaknesses
Conducting relevant literature review

Relevant literature:

Neelu Madan, Andreas Moegelmose, Rajat Modi, Yogesh S. Rawat, Thomas B. Moeslund: Foundation Models for Video Understanding: A Survey (2024), https://arxiv.org/pdf/2405.03770
Xiongkuo Min, Huiyu Duan, Wei Sun, Yucheng Zhu, Guangtao Zhai, Perceptual Video Quality Assessment: A Survey (2024), https://doi.org/10.1007/s11432-024-4133-3

Suitable for Bachelor theses. Contact persons: Srijeet Roy M.Sc.