EResViTNet: Deep Feature Fusion of ResNet and Vision Transformer for Enhanced Classification of Skin Cancer Images

Authors

Keywords:

Vision Transformer, ResNet-50, Feature Fusion, Attention Mechanism, Skin Cancer

Abstract

Skin cancer is among the most common malignancies, and early diagnosis is essential for improving treatment outcomes. However, the visual assessment of dermoscopic images remains subjective and dependent on dermatological expertise. This study proposes EResViTNet, a hybrid deep learning architecture that combines ResNet-50 for local feature extraction with ViT-B/16 for global contextual representation. The resulting feature descriptors are concatenated into a 2816-dimensional vector and subsequently refined using a lightweight channel and spatial attention module, enabling feature recalibration across the joint local and global representation. Image preprocessing includes Error Level Analysis (ELA), resizing, and data augmentation. Evaluated on the nine-class version of the ISIC 2019 dataset, EResViTNet achieved a training accuracy of 98.11% and a held-out test accuracy of 97.38%, averaged across five independent runs, with macro-averaged precision, recall, and F1-score of 97.0%, 98.0%, and 97.0%, respectively. Controlled ablation experiments demonstrated improvements over individual backbones, alternative feature-fusion strategies, and SE, ECA, and CBAM attention mechanisms. The improvements over individual backbones were associated with large effect sizes (Hedges' g > 5). An information-theoretic analysis further examined the complementarity of the learned representations and their relationship to low-rank multimodal fusion. Additional calibration, robustness, and cross-dataset experiments provided further evidence of model stability. PAD-UFES-20 was used as the fully external evaluation dataset, whereas HAM10000 was excluded from external validation because of its overlap with the ISIC 2019 training sources. Grad-CAM visualizations and patch-attention maps provided qualitative insights into the image regions contributing to model predictions. Overall, EResViTNet demonstrates potential as a decision-support approach for dermoscopic image classification, although its clinical utility requires further prospective evaluation involving dermatologists.

Downloads

Published

2026-11-21

How to Cite

Takkella, R., & Kumar, P. P. (2026). EResViTNet: Deep Feature Fusion of ResNet and Vision Transformer for Enhanced Classification of Skin Cancer Images. International Journal of Innovative Technology and Interdisciplinary Sciences, 9(4), 2379–2435. Retrieved from https://journals.tultech.eu/index.php/ijitis/article/view/398