Adaptive Multi-scale Cost Volume Construction and Aggregation for Stereo Matching

doi:10.12068/j.issn.1005-3026.2023.04.001

Abstract

Abstract: Stereo matching based on convolutional neural network has made great progress. Existing methods still suffer from mismatching in weak texture regions， details and edges. Based on the cost volume commonly used in stereo matching， a stereo matching network with adaptive multi-scale cost volume construction and aggregation was proposed. Firstly， the proposed method fully fused the multi-scale features to obtain the recombined features. Then， a learnable feature enhancement module was used to recover the detail information for multi-scale cost volumes. Finally， after intra-scale aggregation based on global attention， an adaptive multi-scale weighting method was proposed for inter-scale aggregation to screen the matching features adapted to the disparity regression of each scale. Massive experiments on the SceneFlow and KITTI2015 datasets show that the proposed method achieves competitive performance with smaller network size which verifies the effectiveness of the proposed method.

Key words: stereo matching; cost volume; feature enhancement; adaptive aggregation

CLC Number:

TP391

PANG Yan-wei， SU Chang， LONG Tao. Adaptive Multi-scale Cost Volume Construction and Aggregation for Stereo Matching[J]. Journal of Northeastern University(Natural Science), 2023, 44(4): 457-468.

References

[1]Zhao Q，Zhou B，Jia L I，et al.A brief survey on virtual reality technology［J］.Science and Technology Review，2016，34(14):71-75.
[2]Chen C Y，Seff A，Kornhauser A，et al.Deepdriving:learning affordance for direct perception in autonomous driving［C］//Proceedings of the IEEE International Conference on Computer Vision.Santiago，2015:2722-2730.
[3]Oisel L，Mémin ，Morin L，et al.One-dimensional dense disparity estimation for three-dimensional reconstruction［J］.IEEE Transactions on Image Processing，2003，12(9):1107-1119.
[4]李晶皎，马利，王爱侠，等.基于改进Patchmatch及切片采样粒子置信度传播的立体匹配算法［J］.东北大学学报(自然科学版)，2016，37(5):609-613.(Li Jing-jiao，Ma Li，Wang Ai-xia，et al.Stereo matching algorithm based on improved Patchmatch and slice sampling particle belief propagation［J］.Journal of Northeastern University(Natural Science)，2016，37(5):609-613.)
[5]张之敏，乔建忠，林树宽，等.一种基于深度网络的视图重建方法［J］.东北大学学报(自然科学版)，2020，41(8):1065-1069.(Zhang Zhi-min，Qiao Jian-zhong，Lin Shu-kuan，et al.A view reconstruction method based on deep network［J］.Journal of Northeastern University(Natural Science)，2020，41(8):1065-1069.)
[6]Chang J R，Chen Y S.Pyramid stereo matching network［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City，2018:5410-5418.
[7]Deng H，Liao Q，Lu Z，et al.Parallax contextual representations for stereo matching［C］//Proceedings of the IEEE International Conference on Image Processing.Anchorage，2021:3193-3197.
[8]Liang Z，Feng Y，Guo Y，et al.Learning for disparity estimation through feature constancy［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City，2018:2811-2820.
[9]Huang Z，Norris T B，Wang P.ES-Net:an efficient stereo matching network［C］//Proceedings of the IEEE International Conference on Intelligent Robots and Systems.Prague，2021.
[10]Yang M，Wu F，Li W.RLStereo:real-time stereo matching based on reinforcement learning［J］.IEEE Transactions on Image Processing，2021，30:9442-9455.
[11]Xu H F，Zhang J Y.AANet:adaptive aggregation network for efficient stereo matching［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Seattle，2020:1956-1965.
[12]He K M，Zhang X Y，Ren S Q，et al.Deep residual learning for image recognition［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016:770-778.
[13]Dai J F，Qi H Z，Xiong Y W，et al.Deformable convolutional networks［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice，2017:764-773.
[14]Kendall A，Martirosyan H，Dasgupta S，et al.End-to-end learning of geometry and context for deep stereo regression［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice，2017:66-75.
[15]Chabra R，Straub J，Sweeny C，et al.StereoDRNet:dilated residual stereo net［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Long Beach，2019:11786-11795.
[16]Mayer N，Ilg E，Hausser P，et al.A large dataset to train convolutional networks for disparity，optical flow，and scene flow estimation［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016:4040-4048.
[17]Menze M，Geiger A.Object scene flow for autonomous vehicles［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston，2015:3061-3070.