数字农科院2.0

DMDC: a cross-attention network for dynamic mask-based dual-camera snapshot hyperspectral Photography

文献类型： 外文期刊

作者： Cai, Zeyu;Zhang, Ziyu;Jin, Chengqian;Da, Feipeng

作者机构：

关键词： Hyperspectral-imaging;RGB;CASSI;Dynamic;Cross-attention

期刊名称： VISUAL COMPUTER

ISSN： 0178-2789

年卷期： 2024 年

页码：

收录情况： SCIE（2024版） ; ; EI（2024版）

摘要： Spectral images can enrich the material information of the reconstructed scene and have an essential role in computer visualization. Coded aperture snapshot spectral imaging (CASSI) for dynamic scenes still faces two problems in reconstruction: 1) The single input limits the network's performance. 2) The spatial light modulator's (SLM) performance has yet to fully develop due to the limitation of fixed mask coding. This paper proposes a cross-attention-based dual-stream network for a dual-camera CASSI system. We argue that RGB images and CASSI measurements are projections of spectral 3D cubes in different 2D spaces and that fusing the spatial features of RGB and the spectral features of CASSI improves the quality of the reconstruction. Upon that, we embed a dynamic mask module in front of the cross-attention-based dual-stream network to further improve the reconstruction quality of the system. Specifically, the dynamic mask module utilizes RGB images to pre-learn the spatial feature distribution of the scene. Then the dynamic mask module guides the SLM in encoding the CASSI. Finally, the RGB and CASSI images are reconstructed using a cross-attention-based dual-stream network to obtain high-quality reconstruction results. Comprehensive experiments on various datasets demonstrate the superior performance of our method. At similar speeds, our method provides a 4.0 dB improvement over existing SOTA methods on clean and noisy datasets. In the snapshot video imaging task, the single snapshot imaging time of DMDC-1stg is less than 50 ms, which verifies the feasibility of our method. (The code has been released at https://github.com/caizeyu1992/DMDC.)

分类号：

  • 相关文献

[1]Reversible-Prior-Based Spectral-Spatial Transformer for Efficient Hyperspectral Image Reconstruction. Zeyu Cai,Zheng Liu,Jian Yu,Ziyu Zhang,Feipeng Da,Chengqian Jin. 2024

[2]棉花早衰程度诊断数码图像数字化指标的研究. 毛树春,王香河,韩迎春,刘伟,王国平,范正义. 2006

[3]不同棉花群体冠层数字图像颜色变化特征研究. 毛树春,韩迎春,冯璐,王国平,范正义,孙恩虹. 2012

[4]基于机器视觉的棉花群体叶绿素监测. 王方永,李少昆,王克如,隋学艳,柏军华,陈兵,刘国庆,谭海珍. 2007

[5]用机器视觉技术获取棉花叶片叶绿素浓度. 王克如,李少昆,王崇桃,杨蕾,谢瑞芝,高世菊,柏军华. 2006

[6]A Machine-Learning Model Based on the Fusion of Spectral and Textural Features from UAV Multi-Sensors to Analyse the Total Nitrogen Content in Winter Wheat. Zongpeng Li,Xinguo Zhou,Qian Cheng,Shuaipeng Fei,Zhen Chen. 2023

[7]Combining UAV multisensor field phenotyping and genome-wide association studies to reveal the genetic basis of plant height in cotton (Gossypium hirsutum). Liqiang Fan,Jiajie Yang,Xuwen Wang,Zhao Liu,Bowei Xu,Li Liu,Chenxu Gao,Xiantao Ai,Fuguang Li,Lei Gao,Yu Yu,Zuoren Yang. 2025

[8]Momentum accelerated unfolding network with spectral–spatial prior for computational spectral imaging. Cai Z.,Li C.,Yu Y.,Jin C.,Da F.. 2024

[9]A MLP architecture fusing RGB and CASSI for computational spectral imaging. Zeyu Cai,Ru Hong,Xun Lin,Jiming Yang,You Liang Ni,Zhen Liu,Chengqian Jin,Feipeng Da. 2024

[10]Remote sensing monitoring the spatiotemporal changes of alpine grassland coverage in the Northern Tibet - art. no. 62982J. Gao Qingzhu,Li Yue,Wan Yunfan,Lin Erda,Sheng Wenping,Yang Kai. 2006

[11]Dynamic root exudation of phenolic allelochemicals from Johnson grass (Sorghum halepense). Huang, Hongjuan,Liu, Chun,Wei, Shouhui,Zhang, Chaoxian,Liu, Chun,Wang, Jinxin.

[12]Understanding the two-stage degradation process of peach gum polysaccharide within ultrasonic field. Jiaxin Chen,Jinfeng Bi,Jingyao Li,Mo Zhou. 2024

[13]Cell-Type Specific miRNA Regulatory Network Responses to ABA Stress Revealed by Time Series Transcriptional Atlases in Arabidopsis. Gao, Zhaoxu,Su, Yanning,Jiao, Guanzhong,Lou, Zhiying,Chang, Le,Yu, Renbo,Xu, Chao,Han, Xue,Wang, Zejia,Li, Jian,Deng, Xing Wang,He, Hang. 2025

作者其他论文 更多>>