Current Issue Cover
时空联合视差优化的立体视频重定向

金康俊, 柴雄力, 邵枫(宁波大学信息科学与工程学院, 宁波 315211)

摘 要
目的 智能适配显示的图像/视频重定向技术近年受到广泛关注。与图像重定向以及2D视频重定向相比,3D视频重定向需要同时考虑视差保持和时域保持。现有的3D视频重定向方法虽然考虑了视差保持却忽略了对视差舒适度的调整,针对因视差过大和视差突变造成视觉不舒适度这一问题,提出了一种基于时空联合视差优化的立体视频重定向方法,将视频视差范围控制在舒适区间。方法 在原始视频上建立均匀网格,并提取显著信息和视差,进而得到每个网格的平均显著值;根据相似性变化原理构建形状保持能量项,利用目标轨迹以及原始视频的视差变化构建时域保持能量项,并结合人眼辐辏调节原理构建视差舒适度调整能量项;结合各个网格的显著性,联合求解所有能量项得到优化后的网格顶点坐标,将其用于确定网格形变,从而生成指定宽高比的视频。结果 实验结果表明,与基于细缝裁剪的立体视频重定向方法对比,本文方法在形状保持、时域保持及视差舒适度方面均具有更好的性能。另外,使用现有的客观质量评价方法对重定向结果进行评价,本文方法客观质量评价指标性能优于均匀缩放和细缝裁剪的视频重定向方法,时间复杂度较低,每帧的时间复杂度至少比细缝裁剪方法降低了98%。结论 提出的时空联合的视差优化方法同时在时域和舒适度上对视差进行优化,并考虑了时域保持,具有良好的视差优化与时域保持效果,展现了较高的稳定性和鲁棒性。本文方法能够用于3D视频的重定向,在保持立体视觉舒适性的同时适配不同尺寸的3D显示屏幕。
关键词
Optimizing spatiotemporal disparities for stereoscopic video retargeting

Jin Kangjun, Chai Xiongli, Shao Feng(Faculty of Information Science and Engineering, Ningbo University, Ningbo 315211, China)

Abstract
Objective In recent years, with the rapid development of the digital video photography, people pay more attention to the imaging quality of videos with the increasing demand for the immersive experience. Therefore, it is a meaningful challenge to adjust stereoscopic content to the required size to accommodate the different resolutions of 3D display devices. Stereoscopic video retargeting is quite different from image retargeting and 2D video retargeting aims to minimize the shape distortions and optimize disparities with temporal coherence in resizing a stereoscopic video. More constraints of 3D video retargeting should be considered, such as, temporal coherence and depth information. However, the existing 3D video retargeting methods usually consider minimizing the depth distortion but not take the visual experience. Many studies show that the unsuitable disparity causes the visual discomfort and visual fatigue. As a result, someone does not have a great experience in 3D video watching. To solve this problem, our study proposes a method to control the disparity into a comfortable range after 3D video retargeting. Our method considers disparity remapping with temporal coherence and angular parallax. Furthermore, retargeting video may shaky and flickering because of the incoherence between frames. Mesh motion trajectory is utilized to constrain the retargeting video for minimizing the shaking. Considering the above factors, a stereoscopic video retargeting method via mesh warping in optimizing spatiotemporal disparities is proposed in this paper. Method First, the mesh in first frame is constructed, and the stereoscopic saliency based on disparity and edge information is estimated. Then, the mean saliency of each grid for mesh importance is computed. Next, the vertex of the mesh in the original video is traced to obtain the vertex trajectories according to the optic flow method. Temporal coherence between the original and retargeted videos can be established by the motion trajectories. Shape distortion is of great importance in affecting the quality of video retargeting. Generally, a high geometric similarity means to a low shape distortion. Therefore, according to significance information, high similarity between the original and deformed grids in those high-significance regions and low similarity in the low-significance regions are maintained. Thus, the shape-preserving energy term is established by using the similarity transformation to minimize the shape distortion. As vergence-accommodation conflict in 3D display may cause fatigue and discomfort, remapping the disparity map into anthropogenic disparity range to reduce the visual fatigue is substantial. After remapping the disparity map, the disparity change with temporal coherence needs to be controlled. In the end, the total energy term is obtained by adding all the energy terms, the optimal grids are obtained by using the linear least square method, and the original video is mapped to obtain the retargeted video. Result Compared with the existing seam carving based stereoscopic video retargeting method, the proposed method achieves a better performance in terms of shape preservation, temporal coherence preservation and disparity remapping energy terms. The objective performance evaluated using the existing objective assessment methods are also higher than the comparison methods. Conclusion A spatiotemporal disparity optimization method, which remaps the video disparity to a comfortable range with temporal coherence, is proposed in this paper. A retargeted video that not only satisfies the viewing comfort but also causes less disparity saltation in time domain can be obtained using this method. A stereoscopic video retargeting method based on grid deformation, which according to the saliency information of the video and a grid deformation equation is established to resize the stereoscopic video into our desired resolution, is proposed in this paper. Results show that the proposed method has an excellent performance in shape preserving, time coherence preserving, and disparity optimization. In the next work, combining the cropping method with the grid deformation method to perform stereoscopic video retargeting and reduce the distortion of the salient target further will be considered. The method proposed in this paper can be used in 3D video retargeting with great stereo visual comfort.
Keywords

订阅号|日报