A Prompt-Preserving MedSAM Enhancement Framework with Historical Feature Fusion and Soft Convolutional Expert Weighting for Medical Image Segmentation

1. Introduction

Medical image segmentation is an important component of biomedical engineering systems for image-guided diagnosis, treatment planning, and quantitative image analysis. Deep segmentation networks have achieved strong performance in many medical imaging tasks, but fully automatic segmentation can be sensitive to target ambiguity, weak boundaries, and domain-specific appearance shifts. Prompt-based segmentation provides a practical alternative by allowing a sparse user input, such as a point or a box, to specify the intended anatomical structure or lesion. The Segment Anything Model (SAM) introduced a general image-encoder, prompt-encoder, and mask-decoder pipeline for segmenting user-specified targets from sparse prompts [1]. MedSAM adapted this promptable interface to medical images through large-scale medical-domain training and has become a representative baseline for SAM-style medical image segmentation [2].

Despite this progress, prompt-based medical image segmentation remains difficult in common two-dimensional imaging scenarios. Fundus images contain nested optic disc and cup structures, which makes accurate cup/disc delineation sensitive to small structural errors [3,4]. Dermoscopic lesion segmentation is challenging because lesion borders are often fuzzy, irregular, and visually heterogeneous [5,6]. Thyroid ultrasound segmentation further suffers from low contrast, speckle noise, and ambiguous nodule margins, which are well-recognized issues in ultrasound segmentation studies [7,8]. Recent attention-based thyroid nodule methods also confirm the importance of region priors and boundary-aware representation [9]. In such settings, the mask decoder must rely on image tokens that preserve small structures, weak edges, and heterogeneous local appearances while still responding to the prompt.

Existing medical adaptations of SAM mainly improve domain adaptation, prompt-conditioned evaluation, or decoder-side adaptation around the original promptable pipeline [2,10,11]. Empirical evaluations have also shown that SAM-style models can behave unevenly across modalities, target types, and prompt settings [12,13]. These findings motivate a complementary encoder-side question: how can promptable medical segmentation retain early structural evidence and adapt local feature enhancement before prompt-conditioned decoding? This question is particularly relevant for biomedical image analysis, where local boundary evidence and small-target structures may be clinically meaningful even when the final task is formulated as a binary mask prediction.

To address this problem, we propose HBF-BCER, a prompt-preserving framework that primarily strengthens MedSAM image-encoder representations. As shown in Figure 1, the framework keeps the original point- and box-prompt interface unchanged, uses the same lightweight decoder-adapter setting across adapted variants, and enhances selected image-encoder blocks with two modules. Historical Branch Fusion (HBF) reinjects cross-layer bottleneck summaries from previous enhanced blocks, making boundary- and small-target-related evidence more accessible to deeper blocks. Balanced Convolutional Expert Routing (BCER) performs soft pixel-wise weighting among heterogeneous convolutional experts, allowing local enhancement to adapt to regions with different textures and boundary conditions. The framework is evaluated on REFUGE2, ISIC2016, and TNMIX under matched simulated-prompt protocols. HBF-BCER improves the matched MedSAM baseline on all three datasets, with REFUGE2 optic cup Dice/IoU increasing from 82.1/73.8 to 91.0/83.7, ISIC2016 Dice/IoU from 87.5/78.6 to 95.9/92.4, and TNMIX Dice/IoU from 81.6/75.1 to 91.4/85.1.

The main contributions of this work are summarized as follows:

• We propose a prompt-preserving MedSAM enhancement framework that keeps the original point- and box-prompt interface unchanged and primarily targets image-encoder representations.
• We design HBF to reuse historical bottleneck features from previous enhanced blocks, improving the preservation of boundary and small-target cues during deep token propagation.
• We design BCER as a soft pixel-wise convolutional expert-weighting module, enabling adaptive local enhancement for heterogeneous medical image regions while avoiding claims of sparse top-k MoE routing.
• We provide matched prompt-based comparisons and ablation analyses on fundus, dermoscopic, and thyroid ultrasound datasets, demonstrating consistent improvements over the matched MedSAM baseline under identical simulated-prompt settings.

2. Related Work

2.1. Promptable Medical Image Segmentation

SAM established a general promptable segmentation paradigm by combining an image encoder, a prompt encoder, and a mask decoder [1]. Subsequent medical adaptations have focused on making this paradigm more suitable for clinical imaging. MedSAM provided a medical-domain SAM adaptation and is widely used as a baseline for prompt-based medical segmentation [2]. Independent evaluations showed that SAM exhibits uneven performance across medical modalities and target types [12]. Subsequent medical SAM studies further emphasized the importance of prompt setting and domain adaptation [13].

Recent medical SAM studies have therefore mainly focused on domain adaptation and prompt-conditioned evaluation around the original SAM interface [2,13]. SAM-Med2D conducts large-scale two-dimensional medical adaptation of SAM [10], while Medical SAM Adapter introduces lightweight medical-domain adapters for parameter-efficient adaptation [11]. More recent SAM2-based work explores image/video-style medical segmentation and stronger foundation-model encoders [14,15,16], and DenseSAM investigates semantic guidance for dense segmentation settings [17]. These studies broaden the SAM-based medical segmentation landscape and motivate comparisons beyond the original MedSAM baseline.

HBF-BCER addresses a different but related problem. It keeps the prompt interface unchanged and uses a consistent lightweight decoder-adapter setting across adapted variants, while the main proposed changes strengthen the image-encoder tokens before prompt-conditioned decoding.

2.2. Structural Preservation in Medical Segmentation

Medical segmentation networks must combine semantic context with fine structural details. U-Net and its skip-connected variants reuse multilevel features to recover spatial details [18,19]. nnU-Net further showed that carefully configured segmentation pipelines can provide strong and robust baselines for biomedical image segmentation [20]. Transformer-based architectures such as TransUNet improve contextual modeling and long-range dependency learning in medical images [21]. Related multi-resolution transformer variants further emphasize the value of scale-aware structural representation [22]. Multi-attention, hierarchical-dependency, UperNet-style ViT, and grouped multi-scale transformer designs also support the role of structural modeling in medical segmentation [23,24,25,26]. These studies highlight the importance of structural preservation and multiscale representation reuse.

Promptable segmentation introduces a different bottleneck. The decoder receives both prompt embeddings and image tokens; therefore, the quality of the image tokens directly affects the final prompt-conditioned mask. HBF follows the structural-preservation motivation of multilevel feature reuse, but implements it inside selected MedSAM image-encoder blocks through historical bottleneck summaries rather than through a conventional encoder-decoder skip path.

2.3. Expert Routing and Adaptive Local Enhancement

Mixture-of-experts and expert-routing mechanisms allocate different inputs to different processing branches, improving model capacity while avoiding a single universal update for all patterns [27]. Recent SAM-related studies also explore MoE-style feature enhancement or shape experts for medical segmentation [28,29]. In medical image analysis, expert or adaptive convolutional mechanisms have been used to model heterogeneous structures and local appearances [30,31]. Although SCESS-Net is developed for audio–visual event localization rather than medical image segmentation, its emphasis on semantic consistency and segment selection is also related to the broader idea of selecting structurally coherent evidence [32]. Such mechanisms are relevant for fundus, dermoscopic, and ultrasound segmentation, where different regions often require different receptive fields and texture responses.

BCER adopts this idea in a lightweight bottleneck-space form. Instead of increasing the whole encoder capacity indiscriminately, it folds bottleneck tokens into a spatial feature map, predicts soft pixel-wise weights over heterogeneous convolutional experts, and projects the enhanced representation back to the token stream. Unlike sparse top-k MoE routing, all experts are softly combined in the reported implementation. The balancing term encourages the experts to remain active and reduces routing collapse.

3. Materials and Methods

3.1. Overall Framework

Figure 1 shows the proposed HBF-BCER framework. Given an input image $I\in {\mathbb{R}}^{B\times 3\times H\times W}$ and a prompt set P, the model follows the standard promptable segmentation pipeline. The normalized image $\tilde{I}$ is transformed into patch tokens and combined with the pretrained image-encoder positional embedding:

$$X^{\left(0\right)}={\varphi }_{\mathrm{patch}}\left(\tilde{I}\right)+\mathcal{I}\left(E_{\mathrm{pos}}^{\mathrm{enc},\mathrm{pre}}\right),$$

(1)

where $\mathcal{I}(·)$ denotes positional-embedding interpolation and reduces to an identity mapping when the input size matches the pretrained configuration. Standard MedSAM encoder layers are denoted by ${\mathcal{T}}_l(·)$. For layers that are not selected for enhancement, the token update is

$$X^{(l+1)}={\mathcal{T}}_l\left(X^{\left(l\right)}\right).$$

(2)

For the selected enhanced layers ${\mathcal{L}}_{\mathrm{enh}}=\{3,6,9,12\}$ under one-based indexing, the standard transformer output is first computed and then refined by HBF and BCER:

$$Y_t={\mathcal{T}}_{l_t}\left(X^{\left(l_t\right)}\right),l_t\in {\mathcal{L}}_{\mathrm{enh}},$$

(3)

$${\overline{X}}_t=Y_t+B_{\mathrm{hbf}}^{\left(l_t\right)}+B_{\mathrm{hist}}^{\left(l_t\right)},$$

(4)

$$X^{(l_t+1)}={\overline{X}}_t+B_{\exp }^{\left(l_t\right)}.$$

(5)

Here $B_{\mathrm{hbf}}^{\left(l_t\right)}$, $B_{\mathrm{hist}}^{\left(l_t\right)}$, and $B_{\exp }^{\left(l_t\right)}$ are the current HBF response, historical bias, and BCER expert bias, respectively. This sequence is consistent with the architectural path in Figure 1: transformer block, HBF, and BCER. The final image tokens are projected as $F_{\mathrm{img}}={\varphi }_{\mathrm{proj}}\left(X^{\left(L\right)}\right)$. The prompt encoder produces sparse and dense prompt embeddings, and the mask decoder predicts low-resolution masks that are subsequently upsampled to the target resolution.

The design is prompt preserving rather than prompt redesigning. The point- and box-prompt interface remains unchanged, and the proposed HBF and BCER modules operate on the image-encoder representation. A lightweight decoder adapter is used as a controlled adaptation component and is kept consistent across adapted variants. Therefore, the main controlled comparison isolates the effect of image-encoder enhancement from prompt-coordinate variation and from changes in the prompt interface.

3.2. Lightweight Decoder Adapter and Adaptation Scope

The proposed novelty lies in HBF and BCER, which are inserted into selected image-encoder blocks. In the implemented adaptation setting, the pretrained ViT-B image encoder is frozen except for the added HBF and BCER parameters. The original prompt encoder and the user-facing point/box prompt interface are preserved. A lightweight decoder adapter is trained to provide a controlled amount of task-specific mask-decoding capacity, but this adapter is not treated as an independent contribution. It is kept identical in all adapted variants used for ablation, including the HBF-only, BCER-only, and full HBF-BCER settings. This clarification distinguishes four settings that should not be conflated: the original MedSAM model, a fully fine-tuned MedSAM reference, a frozen MedSAM model with a lightweight decoder adapter, and the proposed HBF-BCER variant.

3.3. Historical Branch Fusion

Let the t-th enhanced block correspond to encoder layer $l_t\in {\mathcal{L}}_{\mathrm{enh}}$. The standard transformer output $Y_t\in {\mathbb{R}}^{B\times N\times C}$ is first compressed by a bottleneck projection:

$$D_t={\varphi }_{\mathrm{bn}}\left(Y_t\right),D_t\in {\mathbb{R}}^{B\times N_b\times C_b},$$

(6)

where $C_b=C/4$. No spatial downsampling is used in the reported setting; therefore, $N_b=N$ and $H_b=W_b=64$ for a $1024\times 1024$ input with patch size 16. The historical memory stores bottleneck representations from previous enhanced blocks:

$${\mathcal{M}}_t=\{D_{max(1,t-K)},\dots ,D_{t-1}\},K_t=min(K,t-1),$$

(7)

where $K=3$. When $K_t=0$, the historical bias is set to zero. Otherwise, the historical summary is computed as

$$H_{\mathrm{sum},t}={\displaystyle \frac{1}{K_t}}{\displaystyle \sum _{j=max(1,t-K)}^{t-1}}{\varphi }_{\mathrm{align}}\left(D_j\right),$$

(8)

where ${\varphi }_{\mathrm{align}}$ is an identity mapping in the single-scale experiments. The summary is then projected back to the main token dimension:

$$B_{\mathrm{hist}}^{\left(l_t\right)}={\psi }_{\mathrm{up}}\left(H_{\mathrm{sum},t}\right).$$

(9)

Inside the current enhanced block, M lightweight branch transforms operate on $D_t$:

$$R_{t,m}={\tau }_m\left(D_t\right),m=1,\dots ,M.$$

(10)

A learnable non-negative matrix $W_t\in {\mathbb{R}}^{M\times M}$ is normalized by Sinkhorn iterations to obtain a doubly stochastic mixing matrix [33]:

$$S_t=\mathrm{Sinkhorn}\left(W_t\right),{\displaystyle \sum _{q=1}^M}{S}_{t,mq}=1,{\displaystyle \sum _{m=1}^M}{S}_{t,mq}=1.$$

(11)

The mixed response of branch m is

$${\overline{R}}_{t,m}={\displaystyle \sum _{q=1}^M}{S}_{t,mq}{R}_{t,q}.$$

(12)

To avoid reducing the doubly stochastic mixing to a uniform average, the final HBF response uses learnable branch-aggregation weights:

$${\alpha }_t=\mathrm{Softmax}\left(a_t\right),R_{\mathrm{mix},t}={\displaystyle \sum _{m=1}^M}{\alpha }_{t,m}{\overline{R}}_{t,m},$$

(13)

where $a_t\in {\mathbb{R}}^M$ is learned for the t-th enhanced block. This formulation preserves the effect of the Sinkhorn-normalized branch-mixing matrix while allowing the model to emphasize informative mixed branches. The current HBF response is

$$B_{\mathrm{hbf}}^{\left(l_t\right)}={\psi }_{\mathrm{mix}}\left(R_{\mathrm{mix},t}\right).$$

(14)

Thus, HBF provides two complementary terms: a current branch-mixed response $B_{\mathrm{hbf}}^{\left(l_t\right)}$ and a cross-layer historical bias $B_{\mathrm{hist}}^{\left(l_t\right)}$. Figure 2 illustrates the HBF module.

3.4. Balanced Convolutional Expert Routing

BCER enhances local image patterns after HBF refinement. For the t-th enhanced block, the HBF-refined tokens ${\overline{X}}_t$ are projected to the bottleneck space and folded into a spatial feature map:

$$Z_t={\varphi }_{\mathrm{fold}}\left({\varphi }_{\mathrm{bn}}\left({\overline{X}}_t\right)\right)\in {\mathbb{R}}^{B\times C_b\times H_b\times W_b}.$$

(15)

The default implementation uses three convolutional experts: $3\times 3$ and $5\times 5$ depthwise–pointwise experts, and a $1\times 1$ expert with GELU activation. Their outputs are

$$U_{t,e}=f_{t,e}\left(Z_t\right),e=1,\dots ,E.$$

(16)

A lightweight router predicts spatial expert logits $G_t=g_t\left(Z_t\right)$. The soft pixel-wise expert weights are

$$A_t=\mathrm{Softmax}(G_t/{\tau }_{r,t}),A_t\in {\mathbb{R}}^{B\times E\times H_b\times W_b},$$

(17)

where the softmax is applied over the expert dimension and ${\tau }_{r,t}$ is the routing temperature. This operation softly combines all experts at each spatial position rather than selecting a sparse top-1 or top-k expert. The routed representation and expert bias are computed as

$$Z_t^{\prime }={\displaystyle \sum _{e=1}^E}{A}_{t,e}\odot U_{t,e},B_{\exp }^{\left(l_t\right)}={\psi }_{\mathrm{proj}}^{\left(l_t\right)}\left({\varphi }_{\mathrm{unfold}}\left(Z_t^{\prime }\right)\right).$$

(18)

The final enhanced-block output is obtained by adding the expert bias to the HBF-refined tokens:

$$X^{(l_t+1)}={\overline{X}}_t+B_{\exp }^{\left(l_t\right)}.$$

(19)

To prevent a small number of experts from dominating the routing distribution, a load-balancing term is applied. The mean activation of expert e in the t-th enhanced block is

$${\mu }_{t,e}={\displaystyle \frac{1}{B{H}_b{W}_b}}{\displaystyle \sum _{b=1}^B}{\displaystyle \sum _{u=1}^{H_b}}{\displaystyle \sum _{v=1}^{W_b}}{A}_{t,b,e,u,v},$$

(20)

and the balancing loss is

$$L_{\mathrm{bal},t}={\displaystyle \frac{1}{E}}{\displaystyle \sum _{e=1}^E}{\left({\mu }_{t,e}-{\displaystyle \frac{1}{E}}\right)}^2.$$

(21)

Figure 3 shows the BCER module.

3.5. Optimization Objective

HBF-BCER is trained with a segmentation loss and the routing-balancing loss. The segmentation loss combines Dice loss and binary cross-entropy, which is commonly used for class-imbalanced medical segmentation [34,35]. Let $\widehat{M}$ denote the predicted mask and M the ground-truth mask. For channel $c=1,\dots ,C_m$,

$$L_{\mathrm{seg}}={\displaystyle \sum _{c=1}^{C_m}}{\omega }_c\left(L_{\mathrm{Dice}}({\widehat{M}}_c,M_c)+L_{\mathrm{BCE}}({\widehat{M}}_c,M_c)\right).$$

(22)

The total objective is

$$L_{\mathrm{total}}=L_{\mathrm{seg}}+{\lambda }_{\mathrm{bal}}{\displaystyle \sum _{t=1}^T}{L}_{\mathrm{bal},t},$$

(23)

where $T=\left|{\mathcal{L}}_{\mathrm{enh}}\right|$ and ${\lambda }_{\mathrm{bal}}=0.01$.

4. Experiments and Results

4.1. Datasets and Implementation Details

We evaluate HBF-BCER on three two-dimensional medical image segmentation datasets. REFUGE2 is used for optic disc/cup segmentation and follows a fixed 800/400 train/validation split [3,4]. ISIC2016 is used for dermoscopic skin lesion segmentation and follows the official train/test protocol [5,6]. TNMIX is used for thyroid ultrasound nodule segmentation and reflects common ultrasound challenges, including low contrast, speckle noise, and ambiguous nodule margins [7,8,9]. In this work, TNMIX follows a fixed processed benchmark protocol with 3600 images for training and 370 images for evaluation after preprocessing and quality filtering. This setting provides a consistent thyroid-ultrasound test bed for paired comparison between MedSAM and HBF-BCER under identical prompts, preprocessing, and metric computation.

The base promptable model is MedSAM with a SAM ViT-B image encoder and the standard image encoder–prompt encoder–mask decoder pipeline [2]. Unless otherwise stated, HBF-BCER freezes the pretrained image encoder and trains the added HBF and BCER parameters together with the lightweight decoder adapter. The matched MedSAM baseline uses the same prompt protocol, preprocessing, optimizer, and metric pipeline. To avoid ambiguity, “matched MedSAM baseline” in this paper denotes the MedSAM reference under the same simulated prompt and training/evaluation pipeline, whereas “fully fine-tuned MedSAM” is used only for trainable-parameter efficiency framing. We use Adam with an initial learning rate of $1\times {10}^{-4}$, zero weight decay, batch size 2, and a StepLR scheduler with step size 10 and $\gamma =0.5$. Training lasts 40 epochs for REFUGE2, 20 epochs for ISIC2016, and 40 epochs for TNMIX on a single NVIDIA RTX 3090 GPU. The enhanced block set is ${\mathcal{L}}_{\mathrm{enh}}=\{3,6,9,12\}$, the memory length is $K=3$, the branch number is $M=2$, the number of experts is $E=3$, and the routing temperature is initialized to 1 and clipped to [0.1, 10].

Table 1. Definition of MedSAM-related settings used or discussed in this paper. This table separates matched accuracy baselines, ablation controls, and efficiency references.

Setting	Image Encoder	Prompt Encoder	Mask Decoder	Decoder Adapter	HBF	BCER	Main Purpose
Original MedSAM	Pretrained	Pretrained	Original	No	No	No	Reference model definition
Fully fine-tuned MedSAM	Trainable	Task dependent	Trainable	No	No	No	Trainable-parameter efficiency reference
Frozen MedSAM + Adapter	Frozen	Frozen	Adapter-enhanced	Yes	No	No	Adapter-control concept
HBF-only variant	Frozen	Frozen	Adapter-enhanced	Yes	Yes	No	Ablation control
BCER-only variant	Frozen	Frozen	Adapter-enhanced	Yes	No	Yes	Ablation control
HBF-BCER	Frozen	Frozen	Adapter-enhanced	Yes	Yes	Yes	Proposed framework

Table 1. Definition of MedSAM-related settings used or discussed in this paper. This table separates matched accuracy baselines, ablation controls, and efficiency references.

Setting	Image Encoder	Prompt Encoder	Mask Decoder	Decoder Adapter	HBF	BCER	Main Purpose
Original MedSAM	Pretrained	Pretrained	Original	No	No	No	Reference model definition
Fully fine-tuned MedSAM	Trainable	Task dependent	Trainable	No	No	No	Trainable-parameter efficiency reference
Frozen MedSAM + Adapter	Frozen	Frozen	Adapter-enhanced	Yes	No	No	Adapter-control concept
HBF-only variant	Frozen	Frozen	Adapter-enhanced	Yes	Yes	No	Ablation control
BCER-only variant	Frozen	Frozen	Adapter-enhanced	Yes	No	Yes	Ablation control
HBF-BCER	Frozen	Frozen	Adapter-enhanced	Yes	Yes	Yes	Proposed framework

4.2. Prompt Protocol and Metrics

For REFUGE2, disc and cup are predicted as two binary output channels. During training, six positive points are sampled for each image, with three from the disc foreground and three from the cup foreground. A box prompt is enabled with probability $2/3$, and its target overlap is sampled from $\{0.5,0.75\}$. During inference, the main comparison uses a fixed simulated point-prompt protocol without iterative correction. For ISIC2016 and TNMIX, both training and inference use single-target positive point prompts. MedSAM and HBF-BCER receive identical prompt coordinates for paired comparison. This fixed protocol is suitable for isolating representation differences, but it does not fully model real clinical prompt variation; this limitation is discussed explicitly in Section 5.

Dice and Intersection over Union (IoU) are used as the primary overlap metrics [36,37]. REFUGE2 reports disc and cup metrics separately, whereas ISIC2016 and TNMIX report single-target metrics. HD95 is used as a complementary boundary-distance metric and is reported in pixels under the adopted mask resolution [38,39]. Because overlap and boundary-distance metrics capture different error modes, HD95 is interpreted together with Dice/IoU rather than as a redundant measurement [34].

4.3. Matched Prompt-Based Results

Table 2 summarizes the completed matched prompt-based comparison. Under identical simulated prompt inputs, HBF-BCER improves MedSAM on all three datasets. For REFUGE2 optic cup segmentation, Dice/IoU increase from 82.1/73.8 to 91.0/83.7. On ISIC2016, Dice/IoU improve from 87.5/78.6 to 95.9/92.4. On TNMIX, Dice/IoU improve from 81.6/75.1 to 91.4/85.1. These gains are interpreted within each dataset protocol rather than as a single cross-dataset ranking.

4.4. Literature-Reported SAM-Style Reference Results

Recent SAM-style medical segmentation methods are rapidly evolving. To position HBF-BCER with respect to this literature without mixing incompatible protocols, Table 3 summarizes public literature-reported results from Medical SAM 2 on related 2D medical image segmentation benchmarks [15]. These values are not treated as a matched comparison with HBF-BCER because the source paper used its own training, prompt, preprocessing, split, and metric protocols. In particular, the ISIC column in the source corresponds to ISIC 2019 rather than the ISIC2016 protocol used in our main experiments. The same source describes TNMIX as a thyroid-ultrasound benchmark combining 4554 images from TNSCUI and 637 images from DDTI; the 3600/370 setting adopted here is used as a fixed processed split for the controlled MedSAM versus HBF-BCER comparison.

4.5. Non-Prompted Structural References

Table 4 provides structural context using non-prompted segmentation references. These methods do not use prompt inputs and are therefore not matched prompt-based competitors. They are reported only to show how the adopted protocol relates to conventional automatic segmentation architectures. The author-implemented “style” variants denote two-dimensional implementations inspired by the corresponding architectures rather than official challenge submissions. Metrics are computed by the same per-image averaging protocol used in Table 2; therefore, Dice and IoU values should be interpreted as separately averaged metrics rather than converted from a single global confusion matrix.

4.6. Ablation Study

Table 5 evaluates the contributions of HBF and BCER. The row without HBF and without BCER corresponds to the matched MedSAM reference under the same simulated-prompt setting. For the HBF-only, BCER-only, and full variants, the decoder-adapter setting and training protocol are kept unchanged. Therefore, the differences among these rows reflect the encoder-side module switches rather than changes in the decoder configuration.

The single-module variants improve Dice/IoU over the matched reference, while the full HBF-BCER model achieves the best Dice/IoU on all three datasets. REFUGE2 disc HD95 decreases from 35.9 to 6.5. ISIC2016 HD95 worsens for the single-module variants but improves when HBF and BCER are used jointly, indicating that overlap and boundary-distance metrics capture complementary behavior. This pattern supports the design choice of coupling historical structural fusion with adaptive local expert weighting.

4.7. Qualitative, Convergence, and Efficiency Analysis

The qualitative cases in Figure 4, Figure 5, Figure 6 and Figure 7 are consistent with the quantitative improvements. On REFUGE2, HBF-BCER preserves the small optic cup region more consistently under the fixed prompt protocol. On ISIC2016, it produces more complete lesion masks with reduced local leakage near fuzzy boundaries. On TNMIX, it maintains more contiguous nodule regions under low contrast and speckle noise. These visual observations support, but do not by themselves prove, the mechanisms suggested by the ablation results.

Figure 8 reports REFUGE2 convergence behavior. The training loss decreases rapidly in the early epochs, and validation Dice reaches a stable plateau under the adopted training setting. The method also changes the accuracy–efficiency trade-off. With a SAM ViT-B backbone and a $1024\times 1024$ input, fully trainable MedSAM contains 93.74M trainable parameters, 93.74M total parameters, 976.91G FLOPs, and 125.71 ms inference time per image. HBF-BCER contains 17.30M trainable parameters and 111.03M total parameters, with 1039.36G FLOPs and 144.62 ms per image. Thus, it reduces trainable parameters by 81.5%, while increasing total parameters, FLOPs, and inference time by 18.4%, 6.4%, and 15.0%, respectively. This means that the reduction is due to freezing most pretrained parameters rather than to a smaller overall model.

5. Discussion

From a biomedical engineering perspective, the results indicate that preserving structural evidence inside a promptable foundation-model pipeline can improve the reliability of interactive segmentation across several medical imaging modalities. The results support the complementarity of HBF and BCER. HBF is most relevant when early structural cues must be preserved, which is consistent with the large improvement for REFUGE2 optic cup segmentation. BCER is useful when local appearance varies across the image, as in dermoscopic lesions and thyroid ultrasound nodules. The ablation results further show that overlap accuracy and boundary-distance behavior do not always change in the same direction; therefore, HBF-BCER should be interpreted as improving the overall prompt-conditioned segmentation behavior in the evaluated settings rather than as uniformly optimizing every boundary metric in every partial variant.

The comparison with recent SAM-style methods should be interpreted carefully. The public results in Table 3 show that strong medical SAM adaptations such as SAM-Med2D, Medical SAM Adapter, and Medical SAM 2 have become important reference points [10,11,15]. However, those values use different source protocols and are not a substitute for a fully matched rerun. For this reason, our main controlled claim remains the paired improvement over the matched MedSAM baseline under identical prompts and splits. This conservative framing avoids conflating public literature results with author-run matched experiments.

The efficiency profile is also explicit. HBF-BCER reduces trainable parameters relative to full MedSAM fine-tuning, but it increases total parameters, FLOPs, and inference time because the encoder contains additional historical-fusion and soft expert-weighting modules. This trade-off is acceptable for offline and near-interactive segmentation settings, but it should be optimized before deployment in latency-sensitive workflows such as real-time ultrasound.

Several limitations remain. First, the present study focuses on two-dimensional prompt-based segmentation with simulated point and box prompts. Real interactive clinical prompts may be noisier and should be evaluated with repeated prompt sampling, point shifts, box perturbations, and iterative correction. Second, the current evidence is based on three representative two-dimensional datasets; broader validation on additional external datasets and acquisition environments would further clarify generalization. Third, the ViT-B experiments do not fully determine how HBF and BCER scale to larger SAM backbones or SAM2-based architectures. Finally, the current study does not include a full matched rerun of all recent PEFT baselines under the same protocol. These limitations do not invalidate the paired MedSAM comparison, but they define the scope in which the conclusions should be interpreted.

6. Conclusions

Accurate prompt-based medical image segmentation is important for interactive image analysis because clinical targets are often small, weakly contrasted, and dependent on explicit user intent. This study introduced HBF-BCER, a prompt-preserving framework that primarily strengthens MedSAM image-encoder representations while keeping the standard point- and box-prompt interface unchanged. The framework combines Historical Branch Fusion, which reinjects cross-layer structural summaries to preserve boundary- and small-target-related evidence, with Balanced Convolutional Expert Routing, which performs soft pixel-wise weighting among bottleneck convolutional experts to adapt local enhancement to heterogeneous medical appearances. Under matched simulated-prompt protocols, HBF-BCER improved the matched MedSAM baseline on REFUGE2, ISIC2016, and TNMIX: REFUGE2 optic cup Dice/IoU increased from 82.1/73.8 to 91.0/83.7, ISIC2016 Dice/IoU increased from 87.5/78.6 to 95.9/92.4, and TNMIX Dice/IoU increased from 81.6/75.1 to 91.4/85.1. These results suggest that encoder-side historical evidence preservation and adaptive local expert weighting can improve prompt-conditioned segmentation of nested optic structures, irregular dermoscopic lesions, and low-contrast thyroid ultrasound nodules in the evaluated settings. In addition, HBF-BCER reduces the number of trainable parameters relative to full MedSAM fine-tuning, although it introduces additional total parameters, FLOPs, and inference latency. Overall, the proposed framework provides a practical route for improving SAM-style medical segmentation while retaining prompt compatibility; further validation with realistic interactive prompts, matched PEFT baselines, larger multi-center datasets, and three-dimensional imaging scenarios is necessary to clarify its broader clinical generalizability.