Enhancing image quality prediction with self-supervised visual masking

Eurographics 2024

Uğur Çoğalan¹     Mojtaba Bemana¹     Hans-Peter Seidel¹     Karol Myszkowski¹    

1 MPI Informatik

Abstract

Full-reference image quality metrics (FR-IQMs) aim to measure the visual differences between a pair of reference and distorted images, with the goal of accurately predicting human judgments. However, existing FR-IQMs, including traditional ones like PSNR and SSIM and even perceptual ones such as HDR-VDP, LPIPS, and DISTS, still fall short in capturing the complexities and nuances of human perception. In this work, rather than devising a novel IQM model, we seek to improve upon the perceptual quality of existing FR-IQM methods. We achieve this by considering visual masking, an important characteristic of the human visual system that changes its sensitivity to distortions as a function of local image content. Specifically, for a given FR-IQM metric, we propose to predict a visual masking model that modulates reference and distorted images in a way that penalizes the visual errors based on their visibility. Since the ground truth visual masks are difficult to obtain, we demonstrate how they can be derived in a self-supervised manner solely based on mean opinion scores (MOS) collected from an FR-IQM dataset. Our approach results in enhanced FR-IQM metrics that are more in line with human prediction both visually and quantitatively.

Interactive Comparisons

Dataset	Scene	Metric
TID PIPAL LocVis CSIQ	Scene0 Scene1 Scene2 Scene3 Scene4 Scene5 Scene6 Scene7 Scene8 Scene9 Scene10 Scene11	MAE SSIM FLIP VGG

Please click Distorted image to switch between reference!

Distorted image	Reference image

Original	Enhanced

Citation and Materials

Uğur Çoğalan, Mojtaba Bemana, Hans-Peter Seidel, Karol Myszkowski
Enhancing image quality prediction with self-supervised visual masking
Computer Graphics Forum (Proc. EuroGraphics 2024)

@misc{çoğalan2024enhancing,
title={Enhancing image quality prediction with self-supervised visual masking}, 
author={Uğur Çoğalan and Mojtaba Bemana and Hans-Peter Seidel and Karol Myszkowski},
year={2024},
eprint={2305.19858},
archivePrefix={arXiv},
primaryClass={cs.CV}}.

Paper (Full Author's Copy)
			
Github Source Code!