数字农科院2.0

A survey of efficient fine-tuning methods for Vision-Language Models — Prompt and Adapter

文献类型： 外文期刊

作者： Xing J.;Liu J.;Wang J.;Sun L.;Chen X.;Gu X.;Wang Y.

作者机构：

关键词： Adapter;Computer vision;Efficient fine-tuning;Pre-training model;Prompt;Vision-language

期刊名称： Computers and Graphics (Pergamon)

ISSN： 0097-8493

年卷期： 2024 年 119 卷

页码：

收录情况： SCIE（2024版） ; ; EI（2024版）

摘要： Vision Language Model (VLM) is a popular research field located at the fusion of computer vision and natural language processing (NLP). With the emergence of transformer networks and mass web data, numerous large scale VLMs or Vision-Language Pre-training Models (VLPM) have been achieving state-of-the-art results in many tasks, such as retrieval (CLIP) and generation (DALL-E). Although large models have shown impressive results, the cost of retraining and full fine-tuning is prohibitive for general researchers. In recent years, Efficient fine-tuning (EFT) which a very low-cost tuning method has been a good solution to this problem has greatly alleviated this problem, and driven by this, a new fine-tuning paradigm has developed. Since Prompt and Adapter are most widely used in the field of visual language, this review focuses on analysing the progress of the application of these two methods. Firstly, we reviewed the VLM research paradigm based on the differences in pre-training-fine-tuning methods; Next, We categorized the Prompt into 3 types (7 subtypes) of usage patterns based on the different modal information, and categorized the Adapter into 2 types of usage patterns based on whether it plays a role in modal fusion, furthermore we discussed them in vision and vision-language tasks. Finally, we discussed the stability and social ethics of EFT, and possible future research directions were proposed. © 2024 Elsevier Ltd

分类号：

  • 相关文献

[1]Outdoor color rating of sweet cherries using computer vision. Wang, Qi,Wang, Hui,Xie, Lijuan,Zhang, Qin,Wang, Hui,Xie, Lijuan.

[2]Classification of weed seeds based on visual images and deep learning. Tongyun Luo,Jianye Zhao,Yujuan Gu,Shuo Zhang,Xi Qiao,Wen Tian,Yangchun Han. 2023

[3]Feasibility assessment of tree-level flower intensity quantification from UAV RGB imagery: A triennial study in an apple orchard. Chenglong Zhang,João Valente,Wensheng Wang,Leifeng Guo,Aina Tubau Comas,Pieter van Dalfsen,Bert Rijk,Lammert Kooistra. 2023

[4]Maize-IAS: a maize image analysis software using deep learning for high-throughput plant phenotyping. Zhou Shuo,Chai Xiujuan,Yang Zixuan,Wang Hongwu,Yang Chenxue,Sun Tan. 2021

[5]Maize-IAS: a maize image analysis software using deep learning for high-throughput plant phenotyping. Zhou Shuo,Chai Xiujuan,Yang Zixuan,Wang Hongwu,Yang Chenxue,Sun Tan. 2021

[6]Editorial: Remote sensing for field-based crop phenotyping. Jiangang Liu,Zhenjiang Zhou,Bo Li. 2024

[7]SPP-extractor: Automatic phenotype extraction for densely grown soybean plants. Zhou, Wan,Chen, Yijie,Li, Weihao,Zhang, Cong,Xiong, Yajun,Zhan, Wei,Huang, Lan,Wang, Jun,Qiu, Lijuan. 2023

[8]Editorial: Advanced technologies for energy saving, plant quality control and mechanization development in plant factory. Yuxin Tong,Myung Min Oh,Wei Fang. 2023

[9]Vision-based measuring method for individual cow feed intake using depth images and a Siamese network. Xinjie Wang,Baisheng Dai,Xiaoli Wei,Weizheng Shen,Yonggen Zhang,Benhai Xiong. 2023

[10]Accurate recognition of the reproductive development status and prediction of oviposition fecundity in Spodoptera frugiperda (Lepidoptera: Noctuidae) based on computer vision. Chun yang LÜ,Shi shuai GE,Wei HE,Hao wen ZHANG,Xian ming YANG,Bo CHU,Kong ming WU. 2023

[11]Intelligent weight prediction of cows based on semantic segmentation and back propagation neural network. Beibei Xu,Yifan Mao,Wensheng Wang,Guipeng Chen. 2024

[12]Keypoint detection and diameter estimation of cabbage (Brassica oleracea L.) heads under varying occlusion degrees via YOLOv8n-CK network. Jinming Zheng,Xiaochan Wang,Yinyan Shi,Xiaolei Zhang,Yao Wu,Dezhi Wang,Xuekai Huang,Yanxin Wang,Jihao Wang,Jianfei Zhang. 2024

[13]A hybrid method for water stress evaluation of rice with the radiative transfer model and multidimensional imaging. Yufan Zhang,Xiuliang Jin,Liangsheng Shi,Yu Wang,Han Qiao,Yuanyuan Zha. 2025

[14]Lightweight deep learning model for embedded systems efficiently predicts oil and protein content in rapeseed. Mengshuai Guo,Huifang Ma,Xin Lv,Dan Wang,Li Fu,Ping He,Desheng Mei,Hong Chen,Fang Wei. 2025

[15]Recent Advances and Applications of Imaging and Spectroscopy Technologies for Tea Quality Assessment: A Review. Shujun Zhi,Ting An,Han Zhang,Yuhao Bai,Baohua Zhang,Guangzhao Tian. 2025

[16]Dairy farming in the era of artificial intelligence: trend or a real game changer?. Oscar R. Espinoza-Sandoval,Juan Carlos Angeles-Hernandez,Manuel Gonzalez-Ronquillo,Navid Ghavipanje,Naifeng Zhang,A. R. Bayat,Gonzalo Hervás,Ahmed E. Kholif,Marcello Mele,Juan J. Loor,Sokratis Stergiadis,Einar Vargas-Bello-Pérez. 2024

[17]Transfer large models to crop pest recognition—A cross-modal unified framework for parameters efficient fine-tuning. Jianping Liu,Jialu Xing,Guomin Zhou,Jian Wang,Lulu Sun,Xi Chen. 2025

[18]Maize Leaf Area Index Estimation Based on Machine Learning Algorithm and Computer Vision. Fu, Wanna,Chen, Zhen,Cheng, Qian,Li, Yafeng,Zhai, Weiguang,Ding, Fan,Kuang, Xiaohui,Chen, Deshan,Duan, Fuyi. 2025

[19]Innovative integration of computer vision, IoT, and digital twin in food quality and safety assessment. Mengshuai Guo,Xin Lv,Dan Wang,Hong Chen,Fang Wei. 2025

[20]Body parts segmentation and phenotypic traits extraction of pig using an improved point cloud segmentation network with multi-LiDAR. Yehao Jiang,Zechen Li,Jusong Cao,Xuan Li,Xiaoyong Du,Xiong Xiong,Xinyun Li,Haiyan Wang. 2025

作者其他论文 更多>>