Reproducibility and Generalization Analysis of OCAT Net S for Oral Cancer Image Classification

Abstract

Clinical photographs could support earlier referral of suspicious oral lesions, but high internal accuracy does not show whether a model will remain reliable when image source, disease prevalence, or lesion definition changes. This study conducted a secondary experimental analysis of the aggregate measurements reported for OCAT-Net-S by [1]. The analysis examined internal discrimination, component ablation, synthetic perturbation tolerance, computational efficiency, probability calibration, threshold behavior, fold-level dependence, leakage controls, and external transfer. No new patient images or model-training runs were introduced. All new results in this paper are derived from the published tables. OCAT-Net-S achieved an internal AUC-ROC of 0.956, sensitivity of 0.942, and oral-cancer F1-score of 0.942 on 1,238 clinical oral images. Its AUC advantage over TinyViT-5M, the strongest evaluated baseline, was 0.015. The external AUC fell to 0.830 on SMART-OM and 0.840 on the Oral Images Dataset, producing absolute generalization gaps of 0.126 and 0.116. These gaps were 8.40 and 7.73 times larger than the internal advantage over TinyViT-5M. Structural ablations caused a mean AUC loss of 0.046, compared with 0.0128 for training-related ablations. Under the most severe Gaussian-noise condition, OCAT-Net-S retained 96.3% of its clean AUC, compared with 93.8% for TinyViT-5M and 90.3% for ResNet-50. The model used 3.55 million parameters and achieved 0.269 AUC per million parameters, although MobileNetV3-Large remained faster and required fewer operations. Calibration was favorable internally, with a Brier score of 0.0419 and ECE-15 of 0.0174. At an assumed 3% screening prevalence, however, estimated positive predictive value ranged from 0.213 to 0.420 across reported operating points. The findings support the internal technical value of texture-frequency spatial gating and compact local-global feature learning. They also show that distribution shift and prevalence exerted a larger practical effect than the margin separating OCAT-Net-S from its strongest internal comparator. Patient-linked multicenter evaluation, fixed external thresholds, prospective workflow testing, and quantitative lesion localization remain necessary before clinical use.