隐特征监督的孪生网络弱光光流估计
摘 要
目的 弱光照条件下成像存在信噪比低、运动模糊等问题,这对光流估计带来了极大挑战。与现有“先增强—再估计”的光流估计方法不同,为了避免在弱光图像增强阶段损失场景的运动信息,提出一种隐特征监督的弱光光流估计孪生网络学习方法。方法 首先,该方法采用权重共享的孪生网络提取可映射的弱光光流和正常光照光流特征;进而,计算弱光邻帧图像的K近邻相关性卷表,以解决计算4D全对相关性卷表的高时空复杂度问题;在全局运动聚合模块中引入针对二维运动特征的注意力机制,以降低弱光条件下强噪声、运动模糊及低对比度对光流估计的不利影响。最后,提出隐特征监督的光流估计模块,采用正常光照光流特征监督弱光照光流特征的学习,实现高精度的光流估计。结果 与3种最新光流估计方法的对比实验表明,在正常光照条件下,本文方法取得了与现有最佳光流估计方法相近的性能。在FCDN(flying chairs dark noise)数据集上,本文方法光流估计性能最优,相较于次优方法端点误差精度提升了0.16;在多亮度光流估计(various brightness optical flow,VBOF)数据集上,本文方法端点误差精度提升了0.08。结论 本文采用权重共享的双分支孪生网络,实现了对正常光照和弱光照光流特征的准确编码,并采用监督学习方式实现了高精度的弱光照光流估计。实验结果表明,本文方法在弱光光流估计精度及泛化性方面均具有显著优势。本文代码可在https://github.com/suzhansz/LLCV-net.git下载。
关键词
Low-light optical flow estimation with hidden feature supervision using a Siamese network
Xiao Zhaolin1,2, Su Zhan1, Zuo Fengyuan1, Jin Haiyan1,2(1.School of Computer Science and Engineering, Xi'an University of Technology, Xi'an 710048, China;2.Shaanxi Key Laboratory for Network Computing and Security Technology, Xi'an 710048, China) Abstract
Objective Optical flow estimation has been widely used in target tracking,video time-domain super-resolution, behavior recognition,scene depth estimation,and other vision applications. However,imaging under low-light conditions can hardly avoid low signal-to-noise ratio and motion blur,making low-light optical flow estimation very challenging. Applying a pre-stage low-light image enhancement can effectively improve the image visual perception,but it may not be helpful for further optical flow estimation. Unlike the“low light enhancement first and optical flow estimation next”strategy,the low-light image enhancement should be considered with the optical flow estimation simultaneously to prevent the loss of scene motion information. The optical flow features are encoded into the latent space,which enables supervised feature learning for paired low-light and normal-light datasets. This paper also reveals the post-task-oriented feature enhancement outperforms the general visual enhancement of low-light images. The main contributions of this paper can be summarized as follows:1)A dual-branch Siamese network framework is proposed for low-light optical flow estimation. A weightsharing block is used to establish the correlation of motion features between low-light images and normal-light images. 2) An iterative low-light flow estimation module,which can be supervised using normal-light hidden features,is proposed. Our solution is free of explicit enhancement of low-light images. Method This paper proposes a dual-branch Siamese network to encode low-light and the normal-light optical flow features. Then,the encoded features are used to estimate the optical flow in a supervised manner. Our dual-branch feature extractor is constructed using a weight-sharing block,which encodes the motion features. Importantly,our algorithm does not need a pre-stage low-light enhancement,which is usually employed in most existing optical flow estimations. To overcome the high spatial-temporal computational complexity,this paper proposes to compute the K-nearest neighbor correlation volume instead of the 4D all-pair correlation volume. To fuse local and global motion features better,an attention mechanism for the 2D motion feature aggregation is introduced. After the feature extraction,a discriminator is used to distinguish the low-light image features from the normal-light image features. The feature extractor training is completed when the discriminator is incapable to recognize the two. To avoid the explicit enhancement of low-light images,the final optical flow estimation module is composed of a feature enhancement block and a gated recurrent unit(GRU). In an iterative way,the optical flow is decoded from the enhanced feature in the block. A latent feature supervised loss and an iterative similarity loss are used to keep the convergence of the training stage. In the experiment part,the network is trained on an NVIDIA GeForce RTX 3080Ti GPU. The input images are uniformly cropped to 496 × 368 pixels in spatial resolution. Because the low-light and normal-light image paired datasets are limited,the flying chairs dark noise(FCDN)and the various brightness optical flow(VBOF)datasets are jointly used for the model training. Result The proposed algorithm is compared with three state-of-the-art optical flow estimation models on several low-light datasets and normal-light datasets,including FCDN,VBOF,Sintel,and KITTI datasets. Besides the visual comparison,quantitative evaluation with the end-point-error(EPE)metric is conducted. Experimental results show the proposed method achieves a performance comparable with the best available optical flow estimation under normal illumination conditions. The proposed solution improves up to 0. 16 in terms of the EPE index on the FCDN dataset compared with the second-best solution under the low-light condition. On the VBOF dataset,the proposed solution improves 0. 08 in terms of the EPE index compared with the second-best algorithm. Visual comparisons with all the compared methods are also provided. The results show the proposed model preserves more accurate details than other optical flow estimations, especially under low-light conditions. Conclusion In this paper,a dual-branch Siamese network is proposed for realizing the accurate encoding of the optical flow features under normal-light and low-light conditions. The feature extractor is constructed with a weight-sharing block,which enables better-supervised learning for low-light optical flow estimation. The proposed model has remarkable advantages in accuracy and generalizability for the flow estimation. The experimental results indicate the proposed supervised low-light flow estimation outperforms the state-of-the-art solutions in terms of precision.
Keywords
optical flow estimation Siamese network correlation volume global motion aggregation low-light image enhancement
|