A latent diffusion for stable frame interpolation

This article presents a novel approach to video frame interpolation (VFI), called latent diffusion for stable frame interpolation (LD4SFI). LD4SFI leverages a latent diffusion model (LDM) enhanced by a vector-quantized spatiotemporal variational autoencoder (VQ-STVAE). Our method captures intrinsic orthogonal relationships in high-dimensional spatiotemporal data and seamlessly integrates complementary information across diverse video frame sequences. Given the robust sampling capabilities of LDMs, LD4SFI is conditioned on a disparity map that describes the motion dynamics between two neighboring frames. The disparity map is applied to feed explicit spatiotemporal differences into a diffusion model (DM), thereby improving the spatiotemporal smoothness of the interpolation results. With this, LD4SFI efficiently generates interpolated frames, significantly improving the continuity and visual quality of the video content. On UCF-101, densely annotated video segmentation (DAVIS), and SNU-FILM, LD4SFI achieves competitive or improved accuracy relative to existing state-of-the-art (SOTA) methods. LD4SFI produces 0.016 learned perceptual image patch similarity (LPIPS), 36.219 peak signal-to-noise ratio (PSNR), 0.974 structural similarity index (SSIM), 0.031 FloLPIPS, and 20.105 Fréchet inception distance (FID) for the UCF-101 dataset, and 0.072 LPIPS, 30.261 PSNR, 0.912 SSIM, 0.108 FloLPIPS, and 8.037 FID for the DAVIS dataset. Additionally, LD4SFI achieves the highest measured throughput among DMs on a single V100. Experimental results show that LD4SFI outperforms existing SOTA methods, demonstrating highly competitive SOTA performance across standard VFI benchmarks.