360° Structured Light with Learned Metasurfaces

Structured light has proven instrumental in 3D imaging, LiDAR, and holographic light projection. Metasurfaces, comprised of sub-wavelength-sized nanostructures, facilitate 180° field-of-view (FoV) structured light, circumventing the restricted FoV inherent in traditional optics like diffractive optical elements. However, extant metasurface-facilitated structured light exhibits sub-optimal performance in downstream tasks, due to heuristic pattern designs such as periodic dots that do not consider the objectives of the end application. In this work, we present 360° structured light, driven by learned metasurfaces. We propose a differentiable framework, that encompasses a computationally-efficient 180° wave propagation model and a task-specific reconstructor, and exploits both transmission and reflection channels of the metasurface. Leveraging a first-order optimizer within our differentiable framework, we optimize the metasurface design, thereby realizing 360° structured light. We have utilized 360° structured light for holographic light projection and 3D imaging. Specifically, we demonstrate the first 360° light projection of complex patterns, enabled by our propagation model that can be computationally evaluated 50,000x faster than the Rayleigh-Sommerfeld propagation. For 3D imaging, we improve depth-estimation accuracy by 5.09x in RMSE compared to the heuristically-designed structured light. 360° structured light promises robust 360° imaging and display for robotics, extended-reality systems, and human-computer interactions.

Design of 360° Structured Light

We introduce an efficient wave propagation model that accurately simulates light interaction with metasurfaces, achieving over 50,000x speed-up at 512x512 resolution. This model accurately describes 180° full-space wave propagation without paraxial approximation. To achieve 360° structured light, we utilize metasurfaces in both transmission and reflection. We design 360° structured light by learning the metasurface design using a first-order opimizer for downstream tasks.

Holographic image demonstration

As a proof-of-concept of applying 360° structured light, we optimize the phase map of metasurfaces Φ to produce a high-fidelity 360° light projection of a desired holographic image Y by solving the optimization problem which aims to minimize the MSE loss

Learning metasurfaces in End-to-End optimization

We jointly optimize the metasurface phase map and the parameters of a depth-reconstruction neural network, yeilding an optimized 360° structured light suitable for 3D imaging. Belows are the overall framework. It consists of our full-space wave propagation model, differentiable fish-eye image formation and 360 depth reconstructor.

Optimization Progress: Two-stage learning

Our approach to end-to-end optimization for 3D imaging involves two training stages. In the first stages, we simultaneously optimize the metasurface phase map and the network parameters of the depth reconstructor. Once the training of the first stage reaches convergence and the shape of the structured-light pattern stabilizes, we transition to the second stage. In the second stage, we keep the phase map of the metasurface fixed and continue optimizing the reconstructor only.

Our Compact Imaging system for real demonstration

Our imaging system contains four fish-eye cameras and 360 illumination modules. It was mounted on a 3600 x 150mm² breadboard, resulting in a compact configuration.

Metasurface fabrication

Qauntitative Comparison against previous work

Method	iMAE	iRMSE	MAE	RMSE
360° structured light (ours)	0.165	0.319	0.035	0.041
Multi-dot 360° structured light¹	0.523	0.944	0.123	0.209
Passive stereo	0.633	1.08	0.652	0.807

¹ Kim, G., Kim, Y., Yun, J. et al. Metasurface-driven full-space structured light for three-dimensional imaging. Nat Commun 13, 5920 (2022). https://doi.org/10.1038/s41467-022-32117-2

Qualitative comparisons against previous work

The yellow box in the images corresponds to a flat and smooth floor. 360° structured light exploits the learned non-uniform features for robust 3D imaging while the super-cell method¹ and passive stereo struggle in recovering accurate depth.