Assistive artificial intelligence for ultrasound image interpretation in regional anaesthesia: an external validation study

Background Ultrasonound is used to identify anatomical structures during regional anaesthesia and to guide needle insertion and injection of local anaesthetic. ScanNav Anatomy Peripheral Nerve Block (Intelligent Ultrasound, Cardiff, UK) is an artificial intelligence-based device that produces a colour overlay on real-time B-mode ultrasound to highlight anatomical structures of interest. We evaluated the accuracy of the artificial-intelligence colour overlay and its perceived influence on risk of adverse events or block failure. Methods Ultrasound-guided regional anaesthesia experts acquired 720 videos from 40 volunteers (across nine anatomical regions) without using the device. The artificial-intelligence colour overlay was subsequently applied. Three more experts independently reviewed each video (with the original unmodified video) to assess accuracy of the colour overlay in relation to key anatomical structures (true positive/negative and false positive/negative) and the potential for highlighting to modify perceived risk of adverse events (needle trauma to nerves, arteries, pleura, and peritoneum) or block failure. Results The artificial-intelligence models identified the structure of interest in 93.5% of cases (1519/1624), with a false-negative rate of 3.0% (48/1624) and a false-positive rate of 3.5% (57/1624). Highlighting was judged to reduce the risk of unwanted needle trauma to nerves, arteries, pleura, and peritoneum in 62.9–86.4% of cases (302/480 to 345/400), and to increase the risk in 0.0–1.7% (0/160 to 8/480). Risk of block failure was reported to be reduced in 81.3% of scans (585/720) and to be increased in 1.8% (13/720). Conclusions Artificial intelligence-based devices can potentially aid image acquisition and interpretation in ultrasound-guided regional anaesthesia. Further studies are necessary to demonstrate their effectiveness in supporting training and clinical practice. Clinical trial registration NCT04906018.

Conclusions: Artificial intelligence-based devices can potentially aid image acquisition and interpretation in ultrasoundguided regional anaesthesia. Further studies are necessary to demonstrate their effectiveness in supporting training and clinical practice. Clinical trial registration: NCT04906018.

Editor's key points
Ultrasound-guided regional anaesthesia facilitates precision, safety, and effectiveness of peripheral nerve block, but it is technically challenging without advanced training. The use of ScanNav™ (Intelligent Ultrasound, Cardiff, UK), an artificial intelligence-based device that produces a colour overlay on real-time ultrasound images to highlight anatomical structures of interest, was evaluated. Experts reviewed 720 ultrasound videos, with and without ScanNav TM highlighting, to assess accuracy and perceived effect on regional anaesthesia safety and efficacy. The device showed high true-positive/true-negative and low false-positive/false-positive rates in identifying key anatomical structures for the performance of nine peripheral nerve blocks. Further studies are necessary to demonstrate its effectiveness in supporting training and clinical practice.
The use of ultrasound as image guidance for regional anaesthesia was first described in 1989 1 and is now the predominant technique used to guide needle insertion and local anaesthetic deposition. 2 Ultrasound-guided regional anaesthesia (UGRA) can be used to avoid risks associated with general anaesthesia, 3 enhance operating theatre efficiency, and reduce hospital length of stay. 4 Evidence also supports a role in improving outcomes after surgery 4,5 and in mitigating the need for systemic analgesia with dangerous side-effects, such as opioids. 2,4 However, patient access to UGRA can be limited by the availability of a specialist with prerequisite knowledge and skills. 6 Fundamental skills are the acquisition and interpretation of optimal ultrasound images, which involves identification of key sono-anatomical structures. 7 Assistive artificial intelligence (AI) technology could play a role in the future practice of UGRA through supporting ultrasound scanning. 8,9 ScanNav Anatomy Peripheral Nerve Block (Intelligent Ultrasound, Cardiff, UK) uses deep learning based on the U-Net architecture 10 to produce a colour overlay on real-time B-mode ultrasound and highlight structures of interest in UGRA (Fig 1; Supplementary files AeE). The AI models in this system have been trained on more than 800,000 ultrasound images. 11 Training data are presented to the algorithm as paired Example of the colour overlay produced by ScanNav when scanning during a supraclavicular-level brachial plexus block. Blue, first rib; purple, pleura; red, subclavian artery; and yellow, supraclavicular-level brachial plexus nerves (trunks/divisions). unmodified ultrasound image and labelled colour overlay (highlighting the relevant structures on that image). Over time, the algorithm learns to make associations between the labelled region and data in the image. When deployed, it is thus able to make predictions on data in new images and provide a colour overlay on real-time ultrasound. (Further information on the training data is available in Supplementary file F.) It is envisaged that a real-time colour overlay will draw attentional gaze of the operator to the key anatomical structures. Previous work supports the concept that it can aid in acquisition of the correct ultrasound view and correct identification of structures of interest on that view. 12 In this prospective external validation study, experts in UGRA acquired ultrasound scans (without use of ScanNav), and further experts evaluated performance of the AI models. Accuracy of the colour overlay was assessed in relation to key anatomical structures. The perceived potential for highlighting to modify the risk of adverse events (i.e. risk of needle trauma to nerves, arteries, pleura, and peritoneum) and block failure was also evaluated.

Methods
Ethical approval for the collection of ultrasonography scans from healthy volunteers was granted by the Oregon Health & Science University (OHSU) Institutional Review Board (STUDY00022920). The study was registered with ClinicalTrials. gov (NCT04906018).

Ultrasonography scan collection
The process of scan acquisition and review is summarised in Fig  2. Four UGRA experts were recruited from the anaesthesia faculty at OHSU after providing written informed consent. All were board-certified attending anaesthesiologists who had completed advanced training in UGRA (fellowship or equivalent) and regularly use these techniques in their clinical practice.
Forty healthy adult subjects were recruited for scanning after providing written informed consent. Exclusion criteria were age <18 yr and known pathology affecting the areas scanned. Scanning was performed using the SonoSite X-Porte (HFL50xp and L38xp linear probes or C60xp curvilinear probe) and PX (L15e5 and L12-3 linear probes or C5-1 curvilinear probe) ultrasound machines (FUJIFILM SonoSite, Bothell, WA, USA).
A total of 720 scans were performed. For each scan, the scanner stated when they had acquired what they felt to be the optimal view. This frame and the preceding 10 s of the scan were used for later review. Predictive colour overlay, derived by ScanNav TM , was subsequently applied to the ultrasound clips obtained in the acquisition protocol.

Key anatomical structures and adverse events
Nerves, arteries, pleura, and peritoneum were considered as safety-critical structures (although the pleura, in the context of the ESP block, is not typically in view when the needle is inserted, 13 and thus, risk of pneumothorax is low). Target structures for UGRA include peripheral nerves and fascial planes. Therefore, highlighting of the rectus sheath and fascia iliaca and the transverse process of thoracic vertebrae were assessed. The structures for each PNB are detailed in Table 1.

Expert reviewer evaluation
Six additional UGRA experts were recruited for analysis of the highlighting on the recorded scans. Three were based in the USA (board-certified attending anaesthesiologists) and three in Europe (consultant anaesthetist or equivalent). All had completed advanced training in UGRA (fellowship or equivalent) and regularly use these techniques in their clinical practice. Unmodified ultrasound scans and colour-highlighted scan pairs were presented to expert reviewers via an online platform. Videos in the pair played simultaneously with the expert reviewers at liberty to play/pause at their discretion and view multiple times. Scans were labelled with the subject age, sex, and BMI. Three expert reviewers assessed each scan independently: none knew the scans allocated to other expert reviewers or the outcome of their evaluation. A consensus view was taken for each assessment; in cases where no majority was reached, this was classified as 'no consensus'.
For the relevant structures in each scan, reviewers were asked to appraise highlighting accuracy and associated adverse events through the following statements:

Statistical analysis
As this study used a clinical and subjective assessment of the AI models, descriptive statistics of both accuracy and efficacy (perceived influence on risk of adverse event or block failure) have been reported in a manner that reflects clinical use. As all structures for a block region can be present or absent on any single scan, the reported accuracy is presented for each PNB and overall. Accuracy was defined as the sum of the truepositive rate (TPr; TP/total structures) and true-negative rate (TNr; TN/total structures). Rates of false positive (FPr) and false negative (FNr) were similarly calculated but reported independently because of the clinical implications of discriminating between FP and FN.

Discussion
This study is reported according to the Consolidated Standards of Reporting Trials-Artificial Intelligence guidelines. 14 Most prior AI studies of anatomical structure recognition from UGRA images or videos have consisted of data sets from fewer subjects, assessing fewer structures or on fewer videos/ images. Of those published, other than this report, only Gungor and colleagues 15 assessed a commercially available clinical device with clinically relevant endpoints.
We found that ScanNav TM identified anatomical structures essential to safe and efficacious UGRA on real-time ultrasound in 93.5% of cases. The acquisition and interpretation of optimal ultrasound images are fundamental to the practice of UGRA and are a limiting step for non-experts. 3,11 Medical image interpretation is known to be fallible and subjective, even amongst experts. 16e18 Data gathered in this study demonstrate the opportunity to augment human interpretation of ultrasound images during UGRA scanning. The structures highlighted by the AI models closely match those that an international consensus of expert opinion recommends that non-experts identify when performing these procedures. 13 Subjective expert opinion was that highlighting would reduce the risk of recognised complications in 62.9e86.2% of scans. The potential for unintentional needle trauma of a safety critical structure is another limiting factor in the practice of UGRA. Despite the known benefits of UGRA, the majority of patients undergoing surgery amenable to UGRA techniques are not offered a PNB. 6 Such assistive technology has the potential to reduce complications of UGRA and remove a barrier to clinical practice.
Highlighting by ScanNav in this study was perceived to reduce block failure in 81.2% of scans (according to subjective expert opinion). Ultrasound guidance is associated with improved success rates of PNB, faster onset of sensory block, and reduced incidence of vascular injury and local anaesthetic systemic toxicity. 2,19 However, there is still a failure rate to each technique, and the downturn in elective operations conducted during the recent pandemic has led to a commonly held concern over a lack of opportunities to acquire the necessary skills. Medical societies are attempting to promote widespread adoption and standardisation of UGRA. 6,13,20,21 To support the implementation of these aims, innovation is needed to support clinicians in the delivery of safe and efficacious UGRA.
We show that ScanNav TM holds potential to support ultrasound scanning in UGRA and mitigate the risk of complications or block failure. The device has gained regulatory approval for clinical use in Europe (April 2021), and data from this study contribute to evidence submitted for regulatory review in the USA. This and other similar devices could in time support the widespread practice by non-experts or even novices for ultrasound-guided procedures throughout medicine. For example, emergency-department physicians are often familiar with point-of-care ultrasound and interventional procedures, 22 and such assistive technology may aid the practice of UGRA in this setting. Its use for painful interventions currently carried out under sedation obviates the risk of airway compromise, reduces the burden of monitoring, and provides a prolonged pain-free period to facilitate hospital discharge or act as a bridge to definitive treatment (e.g. for hip fractures). Beyond UGRA, use of AI in image interpretation has broader implications across medicine and potentially all of ultrasonography, 23 from screening for developmental dysplasia of the hip 24 to diagnosis of breast cancer. 25 The democratisation of ultrasonography will help ensure that patients have access to the most appropriate interventions, supporting the performance of ultrasound-based interventions by non-experts whilst maintaining relevant clinical standards. 26 The authors recognise limitations to this study. Firstly, our findings must be followed by clinical studies to determine if the predicted benefits are realised in patient outcomes. In particular, use of ultrasound itself has not been shown to reduce the incidence of nerve injury or postoperative neurological symptoms. 2 Assessing the impact of any ultrasound augmentation technology will require rigorous evaluation. Secondly, the remote expert panels reviewing the videos and highlighting were not present when the subject was scanned. Contemporaneous viewing and interpretation of the ultrasound image provide a richer source of information for the operator, and the expert-panel assessments may have been different with this additional knowledge. However, this limitation is attenuated by the fact that three remote experts assessed each video and could play/pause/review them at any point, changing their assessment if required. Multiple practitioners and the luxury of time or changing clinical opinion are often not afforded to physicians in clinical practice. Thirdly, this study is a subjective assessment of the device according to expert opinion. This is particularly true for findings relating to efficacy and safety. Additional studies are in progress to conduct an objective and pixel-by-pixel assessment of AI highlighting accuracy for structure boundaries (compared with expert interpretation). Whilst this will be useful, it should be noted that such assessments may not always correlate with clinical usefulness, and there is a need for measures of performance beyond accuracy. 27 The current assessment has been chosen to be consistent with requirements for reporting device performance for regulatory approval published by the US Food and Drug Administration, 28 which recommends that definitions (e.g. accuracy and FPr/FNr) should be consistent with the intended use of the device. However, it should be noted that there are multiple methods of reporting accuracy of medical devices or tests. 29 Finally, multiple investigators in this study have been involved with the development and regulatory evaluation of this device. The authors hope that, as the technology becomes more widely available, more anaesthetists will engage in detailed study of this and similar devices to determine their true value to our clinical practice.
Whilst further clinical data on patient outcomes are required to confirm the predicted benefits, these data present the case for the accuracy of ScanNav TM and the potential safety and efficacy benefits in UGRA. This study marks a shift in ultrasonography-guided regional anaesthesia, where technological progress is not restricted to image generation but also to image interpretation.