Transfer learning with spinally shared layers
Article
Kabir, H.M. Dipu, Mondal, Subrota Kumar, Alam, Syed Bahauddin and Acharya, U. Rajendra. 2024. "Transfer learning with spinally shared layers." Applied Soft Computing. 163, pp. -. https://doi.org/10.1016/j.asoc.2024.111908
Article Title | Transfer learning with spinally shared layers |
---|---|
ERA Journal ID | 17759 |
Article Category | Article |
Authors | Kabir, H.M. Dipu, Mondal, Subrota Kumar, Alam, Syed Bahauddin and Acharya, U. Rajendra |
Journal Title | Applied Soft Computing |
Journal Citation | 163, pp. - |
Article Number | 111908 |
Number of Pages | 13 |
Year | 2024 |
Publisher | Elsevier |
ISSN | 1568-4946 |
1872-9681 | |
Digital Object Identifier (DOI) | https://doi.org/10.1016/j.asoc.2024.111908 |
Web Address (URL) | https://www.sciencedirect.com/science/article/pii/S1568494624006823 |
Abstract | Transfer-learned models have achieved promising performance in numerous fields. However, high-performing transfer-learned models contain a large number of parameters. In this paper, we propose a transfer learning approach with parameter reduction and potential high performance. Although the high performance depends on the nature of the dataset, we ensure the parameter reduction. In the proposed SpinalNet shared parameters, all intermediate-split-incoming parameters except the first-intermediate-split contain a shared value. Therefore, the SpinalNet shared parameters network contains three parameter groups: (1) first input-split to intermediate-split parameters, (2) shared intermediate-split-incoming parameters, and (3) intermediate-split-to-output-split parameters. The total number of parameters becomes lower than the SpinalNet and traditional fully connected layers due to parameter sharing. Besides the overall accuracy, this paper compares the precision, recall, and F1-score of each class as performance criteria. As a result, both parameter reduction and potential performance improvement become possible for the ResNet-type models, VGG-type traditional models, and Vision Transformers. We applied the proposed model to MNIST, STL-10, and COVID-19 datasets to validate our claims. We also provided a posterior plot of the sample from different models for medical practitioners to understand the uncertainty. Example model training scripts of the proposed model are also shared to GitHub. |
Keywords | COVID; Uncertainty; Transformer; SpinalNet; VGG; ResNet |
Contains Sensitive Content | Does not contain sensitive content |
ANZSRC Field of Research 2020 | 460203. Evolutionary computation |
Byline Affiliations | Charles Sturt University |
Independent Researcher, Australia | |
Macau University of Science and Technology, China | |
University of Illinois Urbana-Champaign, United States | |
National Center for Supercomputing Application, United States | |
Missouri University of Science and Technology, United States | |
School of Mathematics, Physics and Computing | |
Singapore University of Social Sciences (SUSS), Singapore |
Permalink -
https://research.usq.edu.au/item/z8505/transfer-learning-with-spinally-shared-layers
Download files
16
total views8
total downloads1
views this month1
downloads this month