Transformers only look once with nonlinear combination for real-time object detection
Xia, R., Li, G., Huang, Z., Pang, Y. and Qi, M.
Year 2022
Neural Computing and Applications
PublisherSpringer Nature
ISSN 0941-0643
1433-3058
Xia, R., Li, G., Huang, Z., Pang, Y. and Qi, M. 2022. Transformers only look once with nonlinear combination for real-time object detection. Neural Computing and Applications. https://doi.org/10.1007/s00521-022-07333-y
Abstract
In this article, a novel real-time object detector called Transformers Only Look Once (TOLO) is proposed to resolve two problems. The first problem is the inefficiency of building long-distance dependencies among local features for amounts of modern real-time object detectors. The second one is the lack of inductive biases for vision Transformer networks with heavily computational cost. TOLO is composed of Convolutional Neural Network (CNN) backbone, Feature Fusion Neck (FFN), and different Lite Transformer Heads (LTHs), which are used to transfer the inductive biases, supply the extracted features with high-resolution and high-semantic properties, and efficiently mine multiple long-distance dependencies with less memory overhead for detection, respectively.
Moreover, to find the massive potential correct boxes during prediction, we propose a simple and efficient nonlinear combination method between the object confidence and the classification score. Experiments on the PASCAL VOC 2007, 2012, and the MS COCO 2017 datasets demonstrate that TOLO significantly outperforms other state-of-the-art methods with a small input size. Besides, the proposed nonlinear combination method can further elevate the detection performance of TOLO by boosting the results of potential correct predicted boxes without increasing the training process and model parameters.
Digital Object Identifier (DOI) https://doi.org/10.1007/s00521-022-07333-y
References
Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440-1448 https://doi.org/10.1109/ICCV.2015.169 | ||||
Ren S, He K, Girshick R, Sun J (2016) Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137-1149 https://doi.org/10.1109/TPAMI.2016.2577031 PMid:27295650 | ||||
Article | ||||
Google Scholar | ||||
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: single shot multibox detector. In: European conference on computer vision. Springer, pp 21-37 https://doi.org/10.1007/978-3-319-46448-0_2 | ||||
Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 | ||||
Law H, Deng J (2018) Cornernet: Detecting objects as paired keypoints. In: Proceedings of the European conference on computer vision (ECCV), pp 734-750 https://doi.org/10.1007/978-3-030-01264-9_45 | ||||
Duan K, Bai S, Xie L, Qi H, Huang Q, Tian Q (2019) Centernet: keypoint triplets for object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6569-6578 https://doi.org/10.1109/ICCV.2019.00667 | ||||
Tian Z, Shen C, Chen H, He T (2019) Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9627-9636 https://doi.org/10.1109/ICCV.2019.00972 | ||||
Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794-7803 https://doi.org/10.1109/CVPR.2018.00813 | ||||
Luo W, Li Y, Urtasun R, Zemel R (2016) Understanding the effective receptive field in deep convolutional neural networks. In: Proceedings of the 30th international conference on neural information processing systems, pp 4905-4913 | ||||
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132-7141 https://doi.org/10.1109/CVPR.2018.00745 | ||||
Woo S, Park J, Lee Y, Kweon S (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3-19 https://doi.org/10.1007/978-3-030-01234-2_1 | ||||
Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3146-3154 https://doi.org/10.1109/CVPR.2019.00326 | ||||
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998-6008 | ||||
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 | ||||
Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, Jégou H (2021) Training data-efficient image transformers & distillation through attention. In: International conference on machine learning. PMLR, pp 10347-10357 | ||||
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 https://doi.org/10.1109/ICCV48922.2021.00986 | ||||
d'Ascoli S, Touvron H, Leavitt M, Morcos A, Biroli G, Sagun L (2021) Convit: improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697 | ||||
Li Y, Zhang K, Cao J, Timofte R, Van Gool L (2021) Localvit: bringing locality to vision transformers. arXiv preprint arXiv:2104.05707 | ||||
Wang W, Xie E, Li X, Fan DP, Song K, Liang D, Lu T, Luo P, Shao L (2021) Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 https://doi.org/10.1109/ICCV48922.2021.00061 | ||||
Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille AL, Zhou Y (2021) Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 | ||||
Wu S, Li X, Wang X (2020) Iou-aware single-stage object detector for accurate localization. Image Vis Comput 97:103911 https://doi.org/10.1016/j.imavis.2020.103911 | ||||
Article | ||||
Google Scholar | ||||
He Y, Zhang X, Savvides M, Kitani K (2018) Softer-NMS: rethinking bounding box regression for accurate object detection, vol 2, no. 3. arXiv preprint arXiv:1809.08545 https://doi.org/10.1109/CVPR.2019.00300 | ||||
Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303-338 https://doi.org/10.1007/s11263-009-0275-4 | ||||
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: European conference on computer vision. Springer, pp 740-755 https://doi.org/10.1007/978-3-319-10602-1_48 | ||||
Zhang S, Chi C, Yao Y, Lei Z, Li SZ (2020) Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9759-9768 https://doi.org/10.1109/CVPR42600.2020.00978 | ||||
He K, Gkioxari G, P. Dollár, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961-2969 https://doi.org/10.1109/ICCV.2017.322 | ||||
Lin TY, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980-2988 https://doi.org/10.1109/ICCV.2017.324 | ||||
Leng J, Liu Y (2019) An enhanced SSD with feature fusion and visual reasoning for object detection. Neural Comput Appl 31(10):6549-6558 https://doi.org/10.1007/s00521-018-3486-1 | ||||
Lim JS, Astrid M, Yoon HJ, Lee SI (2019) Small object detection using context and attention. arXiv preprint arXiv:1912.06319 | ||||
Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) In: European conference on computer vision. Springer, pp 213-229 https://doi.org/10.1007/978-3-030-58452-8_13 | ||||
Zhu X, Su W, Lu L, Li B, Wang X, Dai J (2020) Deformable detr: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 | ||||
Fang Y, Liao B, Wang X, Fang J, Qi J, Wu R, Niu J, Liu W (2021) You only look at one sequence: rethinking transformer in vision through object detection. arXiv preprint arXiv: 2106.00666 | ||||
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 | ||||
Lin TY, P. Dollár, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117-2125 https://doi.org/10.1109/CVPR.2017.106 | ||||
Dong Y, Cordonnier JB, Loukas A (2021) Attention is not all you need: pure attention loses rank doubly exponentially with depth. arXiv preprint arXiv:2103.03404 | ||||
Zhang Z, He T, Zhang H, Zhang Z, Xie J, Li M (2019) Bag of freebies for training object detection neural networks. arXiv preprint arXiv:1902.04103 | ||||
Touvron H, Cord M, Sablayrolles A, Synnaeve G, Jégou H (2021) Going deeper with image transformers. arXiv preprint arXiv:2103.17239 https://doi.org/10.1109/ICCV48922.2021.00010 | ||||
Kong T, Sun F, Yao A, Liu H, Lu M, Chen Y (2017) Ron: reverse connection with objectness prior networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5936-5944 https://doi.org/10.1109/CVPR.2017.557 | ||||
Fu CY, Liu W, Ranga A, Tyagi A, Berg AC (2017) DSSD: deconvolutional single shot detector. arXiv preprint arXiv:1701.06659 | ||||
Zhang S, Wen L, Bian X, Lei Z, Li SZ (2018) Single-shot refinement neural network for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4203-4212 https://doi.org/10.1109/CVPR.2018.00442 | ||||
Wang K, Lin L, Yan X, Chen Z, Zhang D, Zhang L (2018) Cost-effective object detection: active sample mining with switchable selection criteria. IEEE Trans Neural Netw Learn Syst 30(3):834-850 https://doi.org/10.1109/TNNLS.2018.2852783 PMid:30059324 | ||||
Bell S, Zitnick CL, Bala K, Girshick R (2016) Inside-outside net: detecting objects in context with skip pooling and recurrent neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2874-2883 https://doi.org/10.1109/CVPR.2016.314 | ||||
Shrivastava A, Gupta A, Girshick R (2016) Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 761-769 https://doi.org/10.1109/CVPR.2016.89 | ||||
Kong T, Yao A, Chen Y, Sun F (2016) Hypernet: towards accurate region proposal generation and joint object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 845-853 https://doi.org/10.1109/CVPR.2016.98 | ||||
Liu Z, Du J, Tian F, Wen J (2019) Mr-CNN: a multi-scale region-based convolutional neural network for small traffic sign recognition. IEEE Access 7:57120-57128 https://doi.org/10.1109/ACCESS.2019.2913882 | ||||
Nie J, Anwer RM, Cholakkal H, Khan FS, Pang Y, Shao L (2019) Enriched feature guided refinement network for object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9537-9546 https://doi.org/10.1109/ICCV.2019.00963 | ||||
Zhao Q, Sheng T, Wang Y, Tang Z, Chen Y, Cai L, Ling H (2019) M2det: A single-shot object detector based on multi-level feature pyramid network. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9259-9266 https://doi.org/10.1609/aaai.v33i01.33019259 | ||||
Cao J, Pang Y, Han J, Li X (2019) Hierarchical shot detector. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9705-9714 https://doi.org/10.1109/ICCV.2019.00980 | ||||
Li S, Yang L, Huang J, Hua XS, Zhang L (2019) Dynamic anchor feature selection for single-shot object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6609-6618 https://doi.org/10.1109/ICCV.2019.00671 | ||||
Tan M, Pang R, Le QV (2020) Efficientdet: Scalable and efficient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10781-10790 https://doi.org/10.1109/CVPR42600.2020.01079 | ||||
Wang T, Anwer RM, Cholakkal H, Khan FS, Pang Y, Shao L (2019) Learning rich features at high-speed for single-shot object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1971-1980 https://doi.org/10.1109/ICCV.2019.00206 | ||||
Deng L, Yang M, Li T, He Y, Wang C (2019) Rfbnet: deep multimodal networks with residual fusion blocks for RGB-d semantic segmentation. arXiv preprint arXiv:1907.00135 |
No comments:
Post a Comment