Surgical Instrument Recognition in Laparoscopic Cholecystectomy Using YOLOv8

1. Introduction

Background and Motivation

Over the last few decades, surgical practice has shifted significantly toward Minimally Invasive Surgery (MIS). In the UK alone, laparoscopic operations are performed annually [1,2,3,4], valued for reducing patient trauma, recovery time, and costs relating to the operations [5]. Central to these procedures is an Endoscope; the surgeon operates completely dependent on the video feed from the camera as shown in Figure 1, providing a rich source of intraoperative data [6].

To improve patient safety and address the significant cause of preventable death from medical errors [7], there is a growing need for “Context-Aware Operating Rooms” [8]. Such systems influence the advances in Computer Vision (CV) and Machine Learning (ML) to track workflows and offer real-time decision-making aid [9]. However, the success of these high-level applications depends on one functional building block: the robust and accurate data which recognition of surgical instrument AI algorithms heavily depend on.

Problem Statement

Object detection AI models are highly dependent on the environment in which the video is captured, the quality of the pixels, background, and the sharp edges of the objects within the frames dictated by the model’s robustness and reliability. On the contrary, the surgical domain presents a unique and extreme environment for Deep Learning (DL) algorithms.

While object detection is a well-solved problem for everyday images such as animals, vehicles, fingerprints, and facial expressions, this assumption breaks down when it comes to surgical instruments inside a patient. This is due to the homogeneity of the environment, which is visually monotonous, consisting primarily of red and pink tissues with a low contrast background. Unlike rigid objects, tissues, and organs are constantly in motion and undergo deformation. The instruments are often made of shiny, reflective metals that can cause glare, and specular highlights which confuse the model. Another challenge to overcome is the major visual obstructions, these dynamic events make the tasks more difficult for architectures like YOLO [10]. The visual challenges include; occlusions when using multiple instruments, surgical smoke caused by electrocautery procedures, blood, and fluids, and finally, the reflection of the instruments caused by the bright light of the endoscope.

The research question arising from these profound challenges is to what extends YOLOv8 [11] architecture can be trained to recognise surgical instrument robustly and accurately.

Aims and Objectives

The primary aim of this paper is to utilise YOLOv8 architecture to detect surgical instruments sourced from real laparoscopic cholecystectomy videos in real time. To achieve this goal; first, parsed, cleaned, and converted CholecTrac20 [12] dataset into a standardised format for YOLO training, defined 7 distinct classes as shown in the Table 1. Second, another dataset CholecT50 [13] was consolidated to improve performance of YOLOv8l.

Furthermore, the models were trained and compared all 5 versions of YOLOv8 which are n, s, m, l, and xl to analyse the trade-off between accuracy and speed. Next, an optimisation framework was utilised to perform an automatic hyperparameter tuning to maximise the model’s performance. Finally, a data balancing strategy conducted to re-evaluate the accuracy of the model.

The remainder of this paper is organised as; Section 2 identifies the current research gap and related work, followed by section 3, which details the methodology, including the datasets and YOLOv8 experiments. Section 4 presents the empirical findings and optimisation outcomes, then interprets these results against established benchmark and discuss the study’s limitations, Finally, section 4 summarises the key contributions and directions for future research.

2. Related Work

State-of-Art in Surgical Computer Vision

Early influential studies in surgical instrument recognition utilised two-stage detection architecture such as Faster Region-Based Convolutional Neural Network (Faster R-CNN). Sarikaya et al. [14] demonstrated the potential of DLs using multimodal approach (RGB and Optical Flow), achieving 91% precision on a private dataset. However, the study highlights a significant clinical bottleneck: an inference speed of only 10 FPS, which is insufficient for the real-time demands of live surgery. Furthermore, relying on private datasets (ATLAS Dione) hindered the development of standardised benchmark protocols.

Building on this, Jin et al. [15] shifted the focus towards public benchmarks by applying Faster R-CNN to the M2CAI 2016 challenge dataset [16], while this work contributed a valuable set of public annotations (m2cai 16-instrument-locations) for skill assessment, the results underscored the persistent limitation of two-stage models. Which achieved 63.1% mAP at a rate of only 5 FPS. These early efforts established a critical gap in the literature: the inherent latency prevent such multistage models from deployment in “Context-Aware” operating theatres where immediate feedback is vital.

This study addresses this gap by investigating one-stage detectors capable of balancing high localisation accuracy with real-time frame rates required for clinical applications.

The Adoption of YOLO for Surgical Application

By reframing instrument detection as a single regression problem, the one-stage approach pioneered by YOLO architecture achieves dramatic improvement in speed, making it suitable for real-time video analysis [17]. The integration of YOLO family into surgical computer vision has demonstrated a consistent evolution from simplified tip localisation YOLOv2 [18] to complex, parallel detection-tracking system using YOLOv3 [19].

Subsequent studies utilising YOLOv4 and YOLOv5 achieved high accuracy through custom attention mechanism and loss functions [20,21,22], these results often rely on private, simulated and highly controlled dataset where instruments are photographed on surgical trays or simple monocoloured background rather than during real surgery.

Furthermore, modifying the architecture aimed at boosting precision, such those in YOLOv7x, have occasionally resulted in inference speed 6 to 7 FPS that negate the real-time advantage and clinical use [23]. This persistent gap between controlled experimentation and the real surgical videos marked by smoke, instrument occlusion, and class imbalance emphasise the need for rigorous evaluation of a stable baseline model like YOLOv8 on a public dataset.

Segmentation and Emerging Architecture

Pixel-level segmentation via architectures like U-Net is growing popularity in object detections [24], such models remain vital for medical tasks, however, surgical instrument recognition remains more robustly handled by object detection algorithms that rely on drawing a boundary box around the instruments rather than pixel-level detection. This is due to smoke and motion blur; studies show significant performance gap where detection consistently outperform segmentation in accuracy and speed.

Furthermore, while emerging Transformers models like DETR offer an elegant end-to-end alternative, they currently outperform optimised CNNs like YOLO family in surgical domain, the foundational benchmark for CholecTrack20 dataset confirms YOLO’s dominance but reports a mAP@0.5 of 79.1% using YOLOv8. This baseline provides the primary motivation for this study: to investigate whether YOLOv8 architecture, combined with hyperparameter tuning and data-centric strategies, can surpass existing benchmarks and move towards clinical translation.

3. Methodology

This section outlines a systematic and data-driven methodology for addressing the research question proposed in the previous section. This approach is structured in a pipeline beginning with curation of the dataset, followed by a tournament of all the YOLOv8 variants and two distinct experiments to maximise the performance of the chosen model.

Hardware and Software Implementation

All the codes were written in Python 3.10 programming language using PyTorch framework, training was conducted on a workstation equipped with NVIDIA RTX 3060 GPU with 12GB of VRAM and an Intel core i5 CPU processor, utilising Ultralytics YOLOv8 library with CUDA acceleration.

The Dataset

The primary dataset used in this study is CholecTrack20, which is specialised tracking subset of a larger Cholec80 [25] and CholecT50 surgical datasets. While Cholec80 provides the raw video footage, CholecTrack20 provides the high frequency bounding box annotations required for object detection. To increase the diversity of the training set, we sourced an additional 5 annotated videos from CholecT50 that were not part of the standard CholecTrack20 release, creating a consolidated dataset of 25 unique surgical videos.

The relationship between the datasets is hierarchical: Cholec80 serves as the master set, CholecT50 provides multi-task labels for 50 of those videos, and the CholecTrack20 provides dense tracking data for 20 videos as shown in Figure 2. To prevent temporal leakage, we ensured that frames from the same video were never shared between different splits. The dataset was partitioned by Video ID, ensuring that the model was tested on entirely unseen data. All the videos are high-quality videos from laparoscopic operations performed by 13 surgeons, taken at 25 FPS.

All the videos are high-quality videos from laparoscopic operations performed by 13 surgeons, taken at 25 FPS.

The raw dataset folder as shown in Figure 3 contains CholecTrac20 and CholecT50 folders, labelled as VID01, VID02, ... etc. Each VID folder contains the corresponding JSON files which include the annotation of the instrument for each frame, categories including the instrument, their operations, and video information.

From the raw data, all the JSON files underwent rigorous processing, a Python script was written to automatically go through the JSON files using JSON library. The script read all the JSON files and extracted key information such as instrument visibility within the frame; the visibility flag was set to 1 (instrument is clearly visible), 0 for significantly occluded instruments. This approach was adopted because the first training outcome was low performing and computationally expensive, hence, only the frames with visibility flag = 1 were selected.

Subsequently the script extracted the coordinates of which the instruments were detected, a list of numbers [x-centre, y-centre, box-width, box-height] which identify where exactly the instrument sits in the frame. Finally, the script converted the cleaned and validated annotation data into a format that is required for training YOLO architecture.

YOLOv8 requires a single text file for each corresponding frame containing the class ID and the coordinates of the box around the instrument as shown in Table 2, all the text files were then saved in the Processed data folder.

The conventional method of splitting the training and test data as 80%–20% random frame-based split and standard n-fold cross validation were explicitly rejected to address severe class imbalance and to prevent temporal data leakage which often occur when highly correlated adjacent frames are shared between the training and test sets. Instead, to ensure the model’s ability to generalise to entirely new surgical cases, a custom Python script was written to go through all the 25 videos and count the appearances of each instrument, then we allocated the videos based on how often the rare instruments appear, the videos with high instances of the rare classes were allocated for training. The final partition consisted of 12 videos for training (62.7%) of total instances, 3 videos for validation (10.2%), and 10 videos for testing (27.1%). This specific distribution, detailed in Table 3, was designed to maximise the model’s exposure to rare classes; while maintaining an independent 10 complete surgical videos for testing, this provides a more authentic assessment of clinical reliability than standard random split.

To ensure the reliability of our finding and their compatibility with existing literature, we divided our evaluation into two phases, initially we followed the official CholecTrack20 benchmark protocol, utilising 20 videos with prescribed split ration of 5:1:4. This was to verify our model’s performance against the established 79.1%. Table 4 shows the strategic split for training our model was 6:1:3.

Finally, the processed data split between the images folder which contains test, train, and validation frames, along with the labels folder that contains the annotation for these frames (ground truth) is shown in Figure 4.

Baseline Model Tournament for YOLOv8 Variants

The five distinct size models of YOLOv8 (n, s, m, l, and x). First, the models were trained on strategic split processed data. The training process was set to an image size of 640 x 640 pixels, a batch size of 8 for safe operation and preventing Out of Memory (OOM) error and 50 epoch, then each model’s validation optimised weights (best.pt) checkpoint was evaluated against the unseen test set. Finally, the best performing model (champion model) was selected based on the optimal trade-off between test set mAP and total training time.

Experimental Improvements

After identifying our best performing model, the first experiment conducted on the best performing model was an automated hyperparameter tuning using Genetic Algorithm (GA) integrated into Optuna optimisation framework. The tuning process ran for 25 iterations, and each trial ran over 10 epochs as shown in Figure 5, to identify the optimal configuration of optimisation and augmentation, “the best recipe” for the model to be retrained on.

The second experiment conducted was to close the data imbalance gap. This data-centric approach down-sampled the higher appearance of instruments in the training set. The model was trained on this dataset with substantial copy-pace augmentation to over-sample the rare classes. See the comparison in Figure 12.

Final Evaluation and Model Selection

The definitive edition for this paper was selected by comparing the performance of three candidates, on the training set: The tournament champion, the Balanced model, and the Tuned model; a comprehensive comparison is discussed in the next sections.

4. Results

This section presents the findings of the methodology described, from model selection to the execution of the final training model.

Model Selection Tournament Results

The tournament results to figure out the best performing variant of YOLO v8 were each variant trained for 50 epochs and evaluated on 9,895 images from the test set.

Table 5. a. Tournament validation results for YOLOv8 variants. b. mAP @0.5 for each YOLOv8 variant.

a
Model Variant	Validation mAP@50	Training Time(hrs)	Grasper (Val)	Bipolar (Val)	Hook (Val)	Scissors (Val)	Clipper (Val)	Irrigator (Val)	Spec.bag (Val)
YOLOv8n	67.7%	4.8	88.2%	93.2%	97.4%	33.7%	71.9%	20.5%	68.9%
YOLOv8s	72.4%	5.7	91.2%	89.3%	97.8%	50.0%	75.8%	31.1%	71.4%
YOLOv8m	73.2%	10.1	91.0%	92.6%	97.3%	54.1%	83.5%	18.7%	75.5%
YOLOv8l	73.2%	14.7	91.3%	91.7%	97.1%	50.4%	86.7%	22.4%	73.0%
b
Model Variant			Validation mAP@0.5				Test mAP@0.5
YOLOv8n			67.70%				59.70%
YOLOv8s			72.40%				61.20%
YOLOv8m			73.20%				62.80%
YOLOv8l			73.20%				64.40%

Table 5. a. Tournament validation results for YOLOv8 variants. b. mAP @0.5 for each YOLOv8 variant.

a
Model Variant	Validation mAP@50	Training Time(hrs)	Grasper (Val)	Bipolar (Val)	Hook (Val)	Scissors (Val)	Clipper (Val)	Irrigator (Val)	Spec.bag (Val)
YOLOv8n	67.7%	4.8	88.2%	93.2%	97.4%	33.7%	71.9%	20.5%	68.9%
YOLOv8s	72.4%	5.7	91.2%	89.3%	97.8%	50.0%	75.8%	31.1%	71.4%
YOLOv8m	73.2%	10.1	91.0%	92.6%	97.3%	54.1%	83.5%	18.7%	75.5%
YOLOv8l	73.2%	14.7	91.3%	91.7%	97.1%	50.4%	86.7%	22.4%	73.0%
b
Model Variant			Validation mAP@0.5				Test mAP@0.5
YOLOv8n			67.70%				59.70%
YOLOv8s			72.40%				61.20%
YOLOv8m			73.20%				62.80%
YOLOv8l			73.20%				64.40%

The validation performance for each model is shown in Figure 6a,b.

Overall maP@0.5 of Each Model

Based on the above analysis, the YOLOv8 large variant was selected as the best-performing model, achieving an mAP@0.5 of 64.4% in the testing phase, it provided a near-optimal balance of performance and efficiency, achieving about 99% accuracy of YOLOv8x in two-third of the training time.

Hyperparameter Tuning Result

The Python script for tuning was executed to initiate a 10-epoch-long search for optimal hyperparameters using genetic algorithms and discovered the fittest candidates listed in Table 6.

The final Yolo v8l model was then retrained using the optimal hyperparameters discovered by the tuning process. The training was halted after 100 epochs as the model’s performance plateaued. The final evaluation of this model on the official test yielded an overall mAP@0.5 of 61%. This provides a classic example of overfitting, this critical finding suggests that for this type of datasets, the robust default hyperparameters may be superior to those found by automated tuning on a small validation split.

Data Balancing Experiment Result

The resulting model was evaluated on unseen 10-video test set, yielding a final overall mAP@0.5 of 60.4%, while this headline metric was approximately 4% lower than the baseline model, the per-class analysis reveals the profound success of the method. The model trained on the balanced data achieved noticeably higher AP scores, rare instrument’s performance more than doubled from 24.2% to 51.0% for Scissors and the Clippers increased from 47.8% to 74.5%.

However, despite these gains, the Scissors and Irrigator classes remained significantly below the overall average. This performance gap is primarily attributes to extreme data scarcity, specifically the 34 to 1 instance ratio between Grasper and Scissors shown in Table 3, this limits the model’s ability to learn diverse feature representations.

Furthermore, the intricate geometry of these thin instruments often causes them to fuse with the visually ominous surgical background or be obscured by glares from the endoscope’s camera light. The Irrigator class faces additional challenges from motion blur during rapid cleaning actions, which degrades the sharp edges required for the YOLOv8 architecture to generate high confidence detection, these results demonstrate that while under-sampling is an exceptionally effective technique for improving generalisation across rare classes, as seen in Figure 7, the final performance ceiling is currently dictated by these data-centric challenges rather than model architecture, the per-class mAP@0.5 for all three models shown in Figure 8.

5. Conclusions

This study shows a systematic evaluation of YOLOv8l architecture for surgical instrument detection, achieving 64.4% on a consolidated 25- video dataset, while this performance demonstrates improvement in specific category such as specimen bag, other classes remained challenging compared to previous benchmark. It also highlights the persistent challenge of intraoperative environment. A key finding was the failure of automated hyperparameter tuning, the genetic algorithm overfitted the small validation sample of 3 videos, yielding a reduced test mAP@0.5 of 61%. This suggests that for imbalanced surgical data, robust default configuration may be superior to automated tuning on a small split. Ultimately, this research underscores that the ceiling for surgical computer vision is currently dictates by data-centric challenges rather than architectural limitations. Future progress will rely on the development of high-quality, balanced datasets than hyperparameter refinement.