计算成像前沿进展
顿雄1, 付强2, 李浩天3, 孙天成4, 王建5, 孙启霖6(1.同济大学, 上海 200092;2.阿卜杜拉国王科技大学, 麦加 239556900, 沙特阿拉伯;3.奕目(上海)科技有限公司, 上海 200240;4.Google, 华盛顿州 98103, 美国;5.Snap Research, 纽约 10036, 美国;6.香港中文大学(深圳), 深圳 518172) 摘 要
计算成像是融合光学硬件、图像传感器和算法软件于一体的新一代成像技术,突破了传统成像技术信息获取深度(高动态范围、低照度)、广度(光谱、光场、3维)的瓶颈。本文以计算成像的新设计方法、新算法和应用场景为主线,通过综合国内外文献和相关报道来梳理该领域的主要进展。从端到端光学算法联合设计、高动态范围成像、光场成像、光谱成像、无透镜成像、低照度成像、3维成像和计算摄影等研究方向,重点论述计算成像领域的发展现状、前沿动态、热点问题和趋势。端到端光学算法联合设计包括了可微的衍射光学模型、折射光学模型以及基于可微光线追踪的复杂透镜的模型。高动态范围光学成像从原理到光学调制、多次曝光、多传感器融合以及算法等层面阐述不同方法的优点与缺点以及产业应用。光场成像阐述了基于光场的3维重建技术在超分辨、深度估计和3维尺寸测量等方面国内外的研究进展和产业应用,以及光场在粒子测速及3维火焰重构领域的研究进展。光谱成像阐述了当前多通道滤光片,基于深度学习和波长响应曲线求逆问题,以及衍射光栅、多路复用和超表面等优化实现高光谱的获取。无透镜成像包括平面光学元件的设计和优化,以及图像的高质量重建算法。低照度成像包括低照度情况下基于单帧、多帧、闪光灯和新型传感器的图像噪声去除等。3维成像主要包括针对基于主动方法的深度获取的困难的最新的解决方案,这些困难包括强的环境光干扰(如太阳光)、强的非直接光干扰(如凹面的互反射、雾天的散射)等。计算摄影学是计算成像的一个分支学科,从传统摄影学发展而来,更侧重于使用数字计算的方式进行图像拍摄。在光学镜片的物理尺寸、图像质量受限的情况下,如何使用合理的计算资源,绘制出用户最满意的图像是其主要研究和应用方向。
关键词
Recent progress in computational imaging
Dun Xiong1, Fu Qiang2, Li Haotian3, Sun Tiancheng4, Wang Jian5, Sun Qilin6(1.Tongji University, Shanghai 200092, China;2.King Abdullah University of Science and Technology, Makkah 239556900, Saudi Aribia;3.Yimu(Shanghai)Tech Co. Ltd., Shanghai 200240, China;4.Google Inc., Washington 98103, USA;5.Snap Research, New York 10036, USA;6.The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China) Abstract
Computational imaging breaks the limitation of traditional digital imaging to acquire the information deeper (e.g., high dynamic range imaging and low light imaging) and broader (e.g., spectrum, light field, and 3D imaging). Driven by industry, especially mobile phone manufacturer medical and automotive, computational imaging has become ubiquitous in our daily lives and plays a critical role in accelerating the revolution of industry. It is a new imaging technique that combines illumination, optics, image sensors, and post-processing algorithms. This review takes the latest methods, algorithms, and applications as the mainline and reports the state-of-the-art progress by jointly analyzing the articles and reports at home and aboard. This review covers the topics of end-to-end optics and algorithms design, high dynamic range imaging, light-field imaging, spectrum imaging, lensless imaging, low light imaging, 3D imaging, and computational photography. It focuses on the development status, frontier dynamics, hot issues, and trends in each computational imaging topic. The camera systems have long-term been designed in separated steps:experience-driven lens design followed by costume designed post-processing. Such a general-propose approach achieved success in the past but left the question open for specific tasks and the best compromise among optics, post-processing, and costs. Recent advances aim to build the gap in an end-to-end fashion. To realize the joint optics and algorithms designing, different differentiable optics models have been realized step by step, including the differentiable diffractive optics model, the differentiable refractive optics, and the differentiable complex lens model based on differentiable ray-tracing. Beyond the goal of capturing a sharp and clear image on the sensor, it offers enormous design flexibility that can not only find a compromise between optics and post-processing, but also open up the design space for optical encoding. The end-to-end camera design offers competitive alternatives to modern optics and camera system design. High dynamic range (HDR) imaging has become a commodity imaging technique as evidenced by its applications across many domains, including mobile consumer photography, robotics, drones, surveillance, content capture for display, driver assistance systems, and autonomous driving. This review analyzes the advantages, disadvantages, and industrial applications through analyzing a series of HDR imaging techniques, including optical modulation, multi-exposure, multi-sensor fusion, and post-processing algorithms. Conventional cameras do not record most of the information about the light distribution entering from the world. Light-field imaging records the full 4D light field measuring the amount of light traveling along each ray that intersects the sensor. This review reports how the light field is applied to super-resolution, depth estimation, 3D measurement, and so on and analyzes the state-of-the-art method and industrial application. It also reports the research progress and industrial application in particle image velocimetry and 3D flame imaging. Spectral imaging technique has been used widely and successfully in resource assessment, environmental monitoring, disaster warning, and other remote sensing domains. Spectral image data can be described as a three-dimensional cube. This imaging technique involves capturing the spectrum for each pixel in an image; As a result, the digital images produce detailed characterizations of the scene or object. This review explains multiple methods to acquire spectrum volume data, including the current multi-channel filter, solving the wavelength response curve inversely based on deep learning, diffraction grating, multiplexing, metasurface, and other optimizations to achieve hyper-spectrum acquisition. Lensless imaging eliminates the need for geometric isomorphism between a scene and an image while constructing compact and lightweight imaging systems. It has been applied to bright-field imaging, cytometry, holography, phase recovery, fluorescence, and the quantitative sensing of specific sample properties derived from such images. However, the low reconstructed signal-to-noise ratio makes it an unsolved challenging inverse problem. This review reports the recent progress in designing and optimizing planar optical elements and high-quality image reconstruction algorithms combined with specific applications. Imaging under a low light illumination will be affected by Poisson noise, which becomes increasingly strong as the power of the light source decreases. In the meantime, a series of visual degradation like decreased visibility, intensive noise, and biased color will occur. This review analyzes the challenges of low light imaging and conclude the corresponding solutions, including the noise removal methods of single/multi-frame, flash, and new sensors to deal with the conditions when the sensor exposure to low light. Shape acquisition of three-dimensional objects plays a vital role for various real-world applications, including automotive, machine vision, reverse engineering, industrial inspections, and medical imaging. This review reports the latest active solutions which have been widely applied, including structured light, direct time-of-flight, and indirect time-of-flight. It also notes the difficulties like ambient light (e.g., sunlight), indirect inference (e.g., the mutual reflection of the concave surface, scattering of foggy) of depth acquisition based on those active methods. The use of computation methods in photography refers to digital image capture and processing techniques that use digital calculation instead of optical processes. It can not only improve the camera ability but also add more new features that were not possible at all with traditional film-based photography. Computational photography is an essential branch of computation imaging developed from traditional photography-however, computational photography emphasizes taking a photograph digitally. Limited by the physical size and image quality, computational photography focuses on reasonably arranging the computational resources and showing the high-quality image that pleasures the customer's feeling. As 90 percent of the information transmitted to our human brain is visual, the imaging system plays a vital role for most future intelligence systems. Computational imaging drastically releases human information acquisition ability in no matter depth or scope. For new techniques like metaverse, computational imaging offers a general input tool to collect multi-dimensional visual information for rebuilding the virtual world. This review covers key technological developments, applications, insights, and challenges over the recent years and examines current trends to predict future capabilities.
Keywords
end-to-end camera design high dynamic range imaging light-field imaging spectral imaging lensless imaging low light imaging active 3D imaging computational photography
|