<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.1d1 20130915//EN" "JATS-journalpublishing1.dtd">
<article>
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>No Template</journal-title>
      </journal-title-group>
      <issn publication-format="print"/></journal-meta>
    <article-meta>
      <title-group>
        <article-title>The Value of Radiomic Features of Primary Breast Tumor in STIR Sequences in Predicting Axilla Metastasis</article-title>
      </title-group>
      <contrib-group><contrib contrib-type="author"><name>
            <givenName>Günay</givenName>
            <surname>Rona</surname>
          </name>
          <email>gunayrona@gmail.com</email>
          <xref rid="aff0" ref-type="aff">1</xref>
        </contrib><contrib contrib-type="author"><name>
            <givenName>Meral</givenName>
            <surname>Arifoğlu</surname>
          </name>
          <email/>
          <xref rid="aff0" ref-type="aff">1</xref>
        </contrib><contrib contrib-type="author"><name>
            <givenName>Ahmet Tekin</givenName>
            <surname>Serel</surname>
          </name>
          <email/>
          <xref rid="aff0" ref-type="aff">1</xref>
          <xref rid="aff1" ref-type="aff">2</xref>
        </contrib><aff id="aff0"><institution>Department of Radiology, Kartal Doktor Lütfi Kırdar Training and Research Hospital, University of Health Sciences</institution>
          <addr-line>Istanbul</addr-line><country country="TR">Turkey</country>
        </aff><aff id="aff1"><institution>Department of Urology, Suleyman Demirel University School of Medicine</institution>
          <addr-line>Istanbul</addr-line><country country="TR">Turkey</country>
        </aff></contrib-group><permissions/><abstract>
        <title>Abstract</title>
        <p>Background: Detection of axillary metastases in breast cancer is critical for treatment options and prognosis. The aim of this study is to investigate the value of radiomic features obtained from short tau inversion recovery (STIR) sequences in magnetic resonance imaging (MRI) of primary tumor in breast cancer in predicting axillary lymph node metastasis (ALNM).</p>
        <p>Methods: Lesions of 165 patients with a mean age of 51.12 ±11 (range 28-82) with newly diagnosed invasive breast cancer who underwent breast MRI before treatment were manually segmented from STIR sequences in the 3D Slicer program in all sections. Machine learning (ML) analysis was performed using the extracted 851 features Python 3.11, Pycaret library program. Datasets were randomly divided into train (123, 80%) and independent test (63, 20%) datasets. The performances of ML algorithms were compared with area under curve (AUC), accuracy, recall, presicion and F1 scores.</p>
        <p>Results: Accuracy and AUC in the training set were in the range of 57 %-86 % and 0.50-0.95, respectively. The best model in the training set was the catBoost classifier with an AUC of 0.95 and 84% accuracy. The AUC, accuracy, recall, precision values and F1 score of the CatBoost classifier on the test set were 0.92, 84 %, 89 %, 85 %, 86 %, respectively.</p>
        <p>Conclusion: Radiomic features obtained from primary tumors on STIR sequences have the potential to predict ALNM in invasive breast cancer.</p>
      </abstract>
      <kwd-group>
        <title>Keywords</title>
        <kwd>Breast cancer</kwd>
        <kwd>lymphatic metastasis</kwd>
        <kwd>magnetic resonance imaging</kwd>
        <kwd>Radiomics</kwd>
        <kwd>machine learning</kwd>
      </kwd-group>
      </article-meta>
  </front>
  <body>
    <sec>
      <title>INTRODUCTION</title>
      <p/>
      <p>Axillary lymph node metastasis (ALNM) is one of the most important prognostic factors determining survival in breast cancer. <xref rid="b0" ref-type="bibr">1</xref> The status of the axilla determines the need for axillary lymph node dissection (ALND), axillary radiotherapy, neoadjuvant or adjuvant chemotherapy. <xref rid="b1" ref-type="bibr">2</xref> Accurate determination of the axillary status before treatment is critical in determining individualized treatment options. <xref rid="b2" ref-type="bibr">3</xref> Age, tumor size, tumor quadrant, multifocality, histological grade, pathological type, receptor status, molecular subtype are associated with ALNM. <xref rid="b4" ref-type="bibr">4</xref><xref rid="b5" ref-type="bibr">5</xref><xref rid="b7" ref-type="bibr">6</xref><xref rid="b8" ref-type="bibr">7</xref><xref rid="b9" ref-type="bibr">8</xref><xref rid="b10" ref-type="bibr">9</xref> Radiomic analysis aims to contribute to diagnosis, treatment and follow-up processes by extracting specific quantitative information about diseases that the human eye cannot perceive from medical images. With radiomic analysis, the aim is to maximize the information obtained from images by obtaining quantitative data about signal intensities and spatial distribution of inter-pixel relationships. <xref rid="b11" ref-type="bibr">10</xref> Recently, radiomics has attracted considerable attention in the medical field, especially in oncology.</p>
      <p>Successful results were also obtained in the diagnosis, treatment and classification of breast cancer. <xref rid="b12" ref-type="bibr">11</xref> Also, promising results were acquired in the differentiation of malignant and benign breast masses, in the estimation of the grade and receptor status and subtypes of malignant tumors using radiomic features extracted from magnetic resonance imaging (MRI). This method is also promising in the prediction of neoadjuvant chemotherapy response in breast cancer. In studies performed with MRI, it was found that radiomics successfully helped even in the prediction of recurrence of breast cancer. <xref rid="b13" ref-type="bibr">12</xref><xref rid="b14" ref-type="bibr">13</xref><xref rid="b15" ref-type="bibr">14</xref><xref rid="b16" ref-type="bibr">15</xref><xref rid="b17" ref-type="bibr">16</xref> Radiomics features of axillary lymph nodes from T2W MR images were not successful in predicting ALNM. <xref rid="b18" ref-type="bibr">17</xref> However, ALNM could be predicted with radiomic features obtained from T2-weighted (T2W), diffusion-weighted (DW) and T1+C images of the primary tumor. <xref rid="b19" ref-type="bibr">18</xref><xref rid="b20" ref-type="bibr">19</xref><xref rid="b21" ref-type="bibr">20</xref><xref rid="b22" ref-type="bibr">21</xref><xref rid="b23" ref-type="bibr">22</xref><xref rid="b24" ref-type="bibr">23</xref><xref rid="b25" ref-type="bibr">24</xref> Our aim in this study was to investigate the performance of radiomic features obtained from short tau inversion recovery (STIR) sequences of primary tumor in predicting axillary metastasis.</p>
    </sec>
    <sec>
      <title>METHODS</title>
      <p/>
    </sec>
    <sec>
      <title>Participans</title>
      <p/>
      <p>Patients diagnosed with invasive breast cancer by core biopsy between August 2017 and February 2023 were evaluated retrospectively. Patients who underwent sentinel lymph node biopsy (SLNB) or ALND and patients who underwent MRI before treatment were included in the study. Patients who received neoadjuvant chemotherapy, or had unknown pathology, another malignancy, a recurrent disease, or artifacts in MRI images were excluded from the study. Manual segmentation was performed independently by two radiologists with 8 and 10 years of experience in breast imaging, blinded to the axillary condition of the patients. Segmentation was performed from all axial STIR sequences with tumor. A total of 851 texture features, including first-order, second-order and wavelet-based features were extracted with Slicer-Radiomics (PyRadiomics v3.0.1) <italic>(Figure 1</italic>). One month later, 30 randomly selected patients were independently segmented by the same two radiologists and radiomic features were extracted again. Thus, interobserver agreement was evaluated. Machine Learning Analysis Python 3.11 (Jupyter Notebook, Pycaret Library) was used for data processing and machine learning analysis. The synthetic minority oversampling technique (SMOTE) was used to avoid imbalanced data sets. Data normalization was performed before model develepment.</p>
    </sec>
    <sec>
      <title>MRI Acquisition Protocol</title>
      <p/>
      <p>The data sets were randomly divided into training and independent testing sets. We used 10-fold cross validation of the trained models to avoid data overfitting.</p>
      <p>Overall, 15 machine learning (ML) algorithms were used. The area under the curve (AUC), accuracy, recall, precision and F1 scores were compared with the performances of the ML algorithms. The best model for accuracy and AUC was selected and evaluated on the test set. AUC, accuracy, recall, precision, and F1 scores were given by the confusion matrix. The best model was tuned and finalized. Altogether, 667 features with ICC values above 0.7 were evaluated with ML. Wavelet filtered texture features, maximum 3D diameter, skewness kurtosis and maximum signal features showed a high correlation with ALNM. The features selected by ML algorithms and their importance are presented in <italic>Figure 2</italic>. The accuracy and AUC of ML algorithms on the training set were in the range of 57%-86% and 0.50-0.95, respectively <italic>(Table 1)</italic>. Among the ML algorithms, the best model was CatBoost classifier (AUC:0.95, accuracy: 84%). The ROC curve showing the success of the CatBoost classifier in predicting ALNM is presented in <italic>Figure 3</italic>.    </p>
    </sec>
    <sec>
      <title>Statistical Analysis</title>
      <p/>
    </sec>
    <sec>
      <title>DISCUSSION</title>
      <p/>
      <p>Recently, less invasive treatment approaches for the axilla have been accepted. SLNB is preferred instead of ALND in early stage breast cancer. Determining the condition of the axilla in the preoperative period is important in the development of surgical plans. For individualized and minimally invasive treatment options, it is important to determine the condition of the axilla before treatment. <xref rid="b26" ref-type="bibr">25</xref> Among the imaging methods, ultrasound (US) is the primary tool in the evaluation of the axilla.</p>
      <p>However, approximately 15-20% of patients with negative US findings have metastases in the SLNB. In mammography (MG), 50% of level 1 axillary lymph nodes can be visualized and levels 2 and 3 cannot be evaluated. <xref rid="b27" ref-type="bibr">26</xref> It has been reported that MRI has similar sensitivity to US, but less specificity in detecting nodal metastases. <xref rid="b28" ref-type="bibr">27</xref> In previous studies, ALNM could be predicted by the radiomics features of the primary tumor in MRI. Yu et al., on the other hand, successfully predicted axillary lymph node status on T1+C, T2W, DW MRI with a multiomic signature including radiomics features and clinicopathologic features obtained from lymph nodes and primary tumor. <xref rid="b20" ref-type="bibr">19</xref> Qiu et al. were successful in predicting ALNM using clinicopathological features, morphological features of lymph nodes from MR images, and radiomics features of the primary mass. Radiomic features obtained from DWI, T2W and T1+C images of the primary mass were successful in predicting ALNM with an AUC value of 0.806. <xref rid="b21" ref-type="bibr">20</xref> Wang et al. were able to predict axillary metastases with an AUC value of over 0.80 with the radiomics and deep learning features obtained from dynamic contrast-enhanced (DCE) MRI. <xref rid="b22" ref-type="bibr">21</xref> Chen et al. were able to predict ALNM with the nomogram created by the radiomics and clinicopathological features of the primary tumor obtained from DW and DCE MRI (AUC value in the training set and test set, respectively 0.80, 0.71). <xref rid="b23" ref-type="bibr">22</xref> Using DCE MRI, Liu and colleagues were able to predict ALNM by tumoral and peritumoral radiomics signature (AUC values in the training and test set were 0.872, 0.863, respectively). <xref rid="b24" ref-type="bibr">23</xref> Cui et al. predicted axillary lymph node status with radiomic features obtained from second post-contrast images on MRI with an AUC of 0.86 and an accuracy of 89%. <xref rid="b25" ref-type="bibr">24</xref> However, none of these studies presented the performance of T2W images alone in predicting ALNM.</p>
      <p>Dong and colleagues were successful in predicting sentinel lymph node metastasis with radiomic features obtained from fat-suppressed T2W and DW MRI images. With T2W alone, AUC values were 0.847 in the training set and 0.770 in the validation set. With DW MRI, they obtained AUC values of 0.847 in the training set and 0.787 in the validation set. When they combined the features obtained from T2W and DW images, the AUC values were 0.863 in the training set and 0.805 in the validation set. <xref rid="b19" ref-type="bibr">18</xref> In our study, we predicted ALNM with 0.92 AUC and 84% accuracy in the training and validation set with the radiomic features obtained from STIR sequences by modeling using machine learning analysis. Radiomics features derived from T2W sequences detected ALNM with an AUC of up to 0.85. When DWI and DCE were combined with T2W, the AUC value increased to 0.86. <xref rid="b19" ref-type="bibr">18</xref> In our study, only STIR arrays reached 0.92 AUC. Since STIR sequences are used in routine breast MRI in some centers, they may contribute to the prediction of ALNM. To our knowledge, STIR radiomics have not been used in studies investigating ALNM. STIR sequences stand out compared to other fat suppression techniques by providing more uniform fat suppression without being affected by magnetic field inhomogeneity. Although the signal-to-noise ratio is poor, it is useful because it includes both T1 and T2 contrast.</p>
      <p>There is a correlation between tumor size and lymph node metastasis. <xref rid="b10" ref-type="bibr">9</xref> In our study, the maximum diameter of the tumor and axillary metastases showed a high correlation. Kurtosis and skewness features, which evaluate intralesional homogenetiy, were also correlated with ALNM.</p>
      <p>Our study has some limitations. Its retrospective nature, small number of patients, and choosing the largest lesion in patients with more than one lesion are among the limitations of our study. Another limitation is that we did not divide the patients according to the number of metastatic lymph nodes.</p>
    </sec>
    <sec>
      <title>CONCLUSION</title>
      <p/>
      <p>In conclusion, radiomic features obtained from the primary tumor on STIR sequences have the potential to predict ALNM. STIR sequences are noninvasive and are currently used as a routine component of breast MRI in some centers. However, some lesions cannot be segmented since they have signal intensity close to parchyma in STIR sequences.</p>
    </sec>
    <sec>
      <title>ETHICAL CONSIDERATIONS</title>
      <p/>
      <p>Ethics committee approval was obtained from our institution for this retrospective study (Approve no: 202351425036 ).</p>
    </sec>
    <sec>
      <title>FUNDING</title>
      <p/>
      <p>This research did not receive any funding grants from public, commercial, or nonprofit agencies.</p>
    </sec>
    <sec>
      <fig id="fig_0" orientation="portrait" fig-type="graphic" position="anchor">
        <caption>
          <title>Workflow for extraction of radiomic features from STIR sequences and machine learning analysis</title>
        </caption>
      <graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://upload.wikimedia.org/wikipedia/commons/6/66/SMPTE_Color_Bars.svg"/>
        </fig>
    </sec>
    <sec>
      <fig id="fig_1" orientation="portrait" fig-type="graphic" position="anchor">
        <caption>
          <title>The features selected by the ML algorithms</title>
        </caption>
      <graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://upload.wikimedia.org/wikipedia/commons/6/66/SMPTE_Color_Bars.svg"/>
        </fig>
    </sec>
    <sec>
      <fig id="fig_2" orientation="portrait" fig-type="graphic" position="anchor">
        <caption>
          <title>Receiver operating characteristics curve of the CatBoost classifier in predicting axillary lymph node metastasis.CatBoost classifier was evaluated on the test set. After tuning the AUC, accuracy, recall and precision values and F1 score were 0.92, 84 %, 89 %, 85 %, 86 %, respectively. The CatBoost classifier model had a 92.86% sensitivity and 86.36 % specificity in detecting ALNM. The classification report and confusion matrix showing the performance of the CatBoost model are presented inFigure 4.</title>
        </caption>
      <graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://upload.wikimedia.org/wikipedia/commons/6/66/SMPTE_Color_Bars.svg"/>
        </fig>
    </sec>
    <sec>
      <fig id="fig_3" orientation="portrait" fig-type="graphic" position="anchor">
        <caption>
          <title>Confusion matrix (A), classification report (B) for CatBoost classifier in predicting axillary lymph node metastasis.</title>
        </caption>
      <graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://upload.wikimedia.org/wikipedia/commons/6/66/SMPTE_Color_Bars.svg"/>
        </fig>
    </sec>
    <sec>
      <table-wrap id="tab_0" orientation="portrait">
        <table/>
        <caption>
          <title>Segmentation and Feature Extraction STIR sequences in DCOM format were transferred to the 3D Slicer program (version 4.10.2; https://www.slicer.org ). Resampled images (size:1x1x1 mm) were acquired and normalized.</title>
        </caption>
      </table-wrap>
    </sec>
    <sec>
      <table-wrap id="tab_1" orientation="portrait">
        <table/>
        <caption>
          <title>The data were analyzed by the Statistical Package for Social Sciences (SPSS) version 25.0.0.0 software (IBM Corp., Armonk, N.Y., USA). Percentage, mean, and standard deviation were used to present descriptive results. The one-sample Kolmogorov- Smirnov test was used to see if the groups have a normal distribution. Continuous variables with a normal distribution were shown as mean (± standard deviation [SD]). Interobserver agreement was evaluated using ICC values. Features with an ICC value &gt;0.7 were further checked with ML. RESULTS In this study, 421 patients were evaluated retrospectively. Overall, 106 patients who did not undergo MRI before treatment, 48 patients whose SLNB or ALND data were not available, 52 patients who received neoadjuvant chemotherapy, 6 patients with recurrent disease, 3 patients with concurrent malignancy and 41 patients with artifacts were excluded from the study. Thus, 165 patients with a mean age of 51.12 ±11 (range 28-82) were included in the study. While 92 (55.76 %) patients had axillary metastases, 73 (44.24 %) did not have axillary metastases. The mean lesion size was 24.78 ± 15 (6- 120) mm.</title>
        </caption>
      </table-wrap>
    </sec>
    <sec>
      <table-wrap id="tab_2" orientation="portrait">
        <table/>
        <caption>
          <title>Performance of machine learning models in differentiating those with axillary metastases from those without axillary metastases in patients with invasive breast cancer in the training set from STIR sequences.</title>
        </caption>
      </table-wrap>
    </sec>
    <sec>
      <table-wrap id="tab_3" orientation="portrait">
        <table/>
        <caption>
          <title>The ML models used except Naive Bayes, Dummy Classifier, SVM -Linear Kernel and Ridge Classifier showed successful performance with at least 0.73 AUC and 74 % accuracy. CatBoost Classifier, Extra Trees Classifier, Light Gradient Boosting Machine, Gradient Boosting Classifier and Extreme Gradient Boosting had AUC values above 0.93 and accuarrcy values above 86 %. Among the models CatBoost Classifier, Extra Trees Classifier, Light Gradient Boosting, Gradient Boosting Classifier and Random Forest Classifier had the highest performance with 0.95 AUC values and with 84-86 % accurracy values. Among these models, CatBoost Classifier had the highest recall and precision values with 0.93 and 82 %, respectively.</title>
        </caption>
      </table-wrap>
    </sec>
  </body>
  <back>
    <ref-list>
      <title>References</title><ref id="b27">
        <element-citation publication-type="journal">
          <article-title>Axillary Nodal Evaluation in Breast Cancer: State of the Art</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>J M</given-names>
              <surname>Chang</surname>
            </name>
            <name>
              <given-names>Jwt</given-names>
              <surname>Leung</surname>
            </name>
            <name>
              <given-names>L</given-names>
              <surname>Moy</surname>
            </name>
            <name>
              <given-names>S M</given-names>
              <surname>Ha</surname>
            </name>
            <name>
              <given-names>W K</given-names>
              <surname>Moon</surname>
            </name>
          </person-group>
          <source>Radiology</source>
          <volume>295</volume>
          <fpage>500</fpage>
          <lpage>515</lpage>
          <year>2020</year>
        </element-citation>
        </ref>
      <ref id="b15">
        <element-citation publication-type="journal">
          <article-title>Prediction of malignancy by a radiomic signature from contrast agent-free diffusion MRI in suspicious breast lesions found on screening mammography</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>S</given-names>
              <surname>Bickelhaupt</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Paech</surname>
            </name>
            <name>
              <given-names>P</given-names>
              <surname>Kickingereder</surname>
            </name>
            <name>
              <given-names>F</given-names>
              <surname>Steudle</surname>
            </name>
            <name>
              <given-names>W</given-names>
              <surname>Lederer</surname>
            </name>
            <name>
              <given-names>H</given-names>
              <surname>Daniel</surname>
            </name>
          </person-group>
          <source>J Magn Reson Imaging</source>
          <volume>46</volume>
          <fpage>604</fpage>
          <lpage>616</lpage>
          <year>2017</year>
        </element-citation>
        </ref>
      <ref id="b6">
        <element-citation publication-type="journal">
          <article-title/>
          <source>Ann Surg Oncol</source>
          <volume>10</volume>
          <issue>10</issue>
          <fpage>1140</fpage>
          <lpage>51</lpage>
          <year>2003</year>
        </element-citation>
        </ref>
      <ref id="b17">
        <element-citation publication-type="journal">
          <article-title>A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>N</given-names>
              <surname>Antropova</surname>
            </name>
            <name>
              <given-names>B Q</given-names>
              <surname>Huynh</surname>
            </name>
            <name>
              <given-names>M L</given-names>
              <surname>Giger</surname>
            </name>
          </person-group>
          <source>Med Phys</source>
          <volume>44</volume>
          <fpage>5162</fpage>
          <lpage>5171</lpage>
          <year>2017</year>
        </element-citation>
        </ref>
      <ref id="b1">
        <element-citation publication-type="journal">
          <article-title>Axillary node interventions in breast cancer: a systematic review</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>R</given-names>
              <surname>Rao</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Euhus</surname>
            </name>
            <name>
              <given-names>H G</given-names>
              <surname>Mayo</surname>
            </name>
            <name>
              <given-names>C</given-names>
              <surname>Balch</surname>
            </name>
          </person-group>
          <source>JAMA</source>
          <volume>310</volume>
          <fpage>1385</fpage>
          <lpage>1394</lpage>
          <year>2013</year>
        </element-citation>
        </ref>
      <ref id="b7">
        <element-citation publication-type="journal">
          <article-title>A nomogram to predict the probability of axillary lymph node metastasis in female patients with breast cancer in China: A nationwide, multicenter, 10-year epidemiological study</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>J</given-names>
              <surname>Zhang</surname>
            </name>
            <name>
              <given-names>X</given-names>
              <surname>Li</surname>
            </name>
            <name>
              <given-names>R</given-names>
              <surname>Huang</surname>
            </name>
            <name>
              <given-names>W L</given-names>
              <surname>Feng</surname>
            </name>
            <name>
              <given-names>Y N</given-names>
              <surname>Kong</surname>
            </name>
            <name>
              <given-names>F</given-names>
              <surname>Xu</surname>
            </name>
          </person-group>
          <source>Oncotarget</source>
          <volume>8</volume>
          <fpage>35311</fpage>
          <lpage>35325</lpage>
          <year>2017</year>
        </element-citation>
        </ref>
      <ref id="b23">
        <element-citation publication-type="journal">
          <article-title>Deep Learning Radiomics of Preoperative Breast MRI for Prediction of Axillary Lymph Node Metastasis in Breast Cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>Y</given-names>
              <surname>Chen</surname>
            </name>
            <name>
              <given-names>L</given-names>
              <surname>Wang</surname>
            </name>
            <name>
              <given-names>X</given-names>
              <surname>Dong</surname>
            </name>
            <name>
              <given-names>R</given-names>
              <surname>Luo</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Ge</surname>
            </name>
            <name>
              <given-names>H</given-names>
              <surname>Liu</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Zhang</surname>
            </name>
          </person-group>
          <source>J Digit Imaging</source>
          <volume>1</volume>
          <issue>9</issue>
          <year>2023</year>
        </element-citation>
        </ref>
      <ref id="b3">
        <element-citation publication-type="misc">
          <volume>149</volume>
          <fpage>3457</fpage>
          <lpage>3467</lpage>
        </element-citation>
        </ref>
      <ref id="b14">
        <element-citation publication-type="journal">
          <article-title>Additive Benefit of Radiomics Over Size Alone in the Distinction Between Benign Lesions and Luminal A Cancers on a Large Clinical Breast MRI Dataset</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>H M</given-names>
              <surname>Whitney</surname>
            </name>
            <name>
              <given-names>N S</given-names>
              <surname>Taylor</surname>
            </name>
            <name>
              <given-names>K</given-names>
              <surname>Drukker</surname>
            </name>
            <name>
              <given-names>A V</given-names>
              <surname>Edwards</surname>
            </name>
            <name>
              <given-names>J</given-names>
              <surname>Papaioannou</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Schacht</surname>
            </name>
          </person-group>
          <source>Acad Radiol</source>
          <volume>26</volume>
          <fpage>202</fpage>
          <lpage>209</lpage>
          <year>2019</year>
        </element-citation>
        </ref>
      <ref id="b4">
        <element-citation publication-type="journal">
          <article-title>Frequency and predictors of axillary lymph node metastases in invasive breast cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>B</given-names>
              <surname>Chua</surname>
            </name>
            <name>
              <given-names>O</given-names>
              <surname>Ung</surname>
            </name>
            <name>
              <given-names>R</given-names>
              <surname>Taylor</surname>
            </name>
            <name>
              <given-names>J</given-names>
              <surname>Boyages</surname>
            </name>
          </person-group>
          <source>ANZ J Surg</source>
          <volume>71</volume>
          <issue>12</issue>
          <fpage>723</fpage>
          <lpage>731</lpage>
          <year>2001</year>
        </element-citation>
        </ref>
      <ref id="b20">
        <element-citation publication-type="journal">
          <article-title>Magnetic resonance imaging radiomics predicts preoperative axillary lymph node metastasis to support surgical decisions and is associated with tumor microenvironment in invasive breast cancer: A machine learning, multicenter study</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>Y</given-names>
              <surname>Yu</surname>
            </name>
            <name>
              <given-names>Z</given-names>
              <surname>He</surname>
            </name>
            <name>
              <given-names>J</given-names>
              <surname>Ouyang</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Tan</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Chen</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Gu</surname>
            </name>
          </person-group>
          <source>EBioMedicine</source>
          <volume>69</volume>
          <fpage>103460</fpage>
          <lpage>103460</lpage>
          <year>2021</year>
        </element-citation>
        </ref>
      <ref id="b12">
        <element-citation publication-type="journal">
          <article-title>Radiomics in breast cancer classification and prediction</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>A</given-names>
              <surname>Conti</surname>
            </name>
            <name>
              <given-names>A</given-names>
              <surname>Duggento</surname>
            </name>
            <name>
              <given-names>I</given-names>
              <surname>Indovina</surname>
            </name>
            <name>
              <given-names>M</given-names>
              <surname>Guerrisi</surname>
            </name>
            <name>
              <given-names>N</given-names>
              <surname>Toschi</surname>
            </name>
          </person-group>
          <source>Semin Cancer Biol</source>
          <volume>72</volume>
          <fpage>238</fpage>
          <lpage>250</lpage>
          <year>2021</year>
        </element-citation>
        </ref>
      <ref id="b13">
        <element-citation publication-type="journal">
          <article-title>Integrated radiomic framework for breast cancer and tumor biology using advanced machine learning and multiparametric</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>V S</given-names>
              <surname>Parekh</surname>
            </name>
            <name>
              <given-names>M A</given-names>
              <surname>Jacobs</surname>
            </name>
          </person-group>
          <source>MRI NPJ Breast Cancer</source>
          <volume>3</volume>
          <fpage>43</fpage>
          <lpage>43</lpage>
          <year>2017</year>
        </element-citation>
        </ref>
      <ref id="b25">
        <element-citation publication-type="journal">
          <article-title>Preoperative Prediction of Axillary Lymph Node Metastasis in Breast Cancer using Radiomics Features of DCE-MRI</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>X</given-names>
              <surname>Cui</surname>
            </name>
            <name>
              <given-names>N</given-names>
              <surname>Wang</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Zhao</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Chen</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Li</surname>
            </name>
            <name>
              <given-names>M</given-names>
              <surname>Xu</surname>
            </name>
          </person-group>
          <source>Sci Rep</source>
          <volume>9</volume>
          <issue>1</issue>
          <fpage>2240</fpage>
          <lpage>2240</lpage>
          <year>2019</year>
        </element-citation>
        </ref>
      <ref id="b11">
        <element-citation publication-type="journal">
          <article-title>Radiomics in medical imaging-"how-to" guide and critical reflection</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>J E</given-names>
              <surname>Van Timmeren</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Cester</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Tanadini-Lang</surname>
            </name>
            <name>
              <given-names>H</given-names>
              <surname>Alkadhi</surname>
            </name>
            <name>
              <given-names>B</given-names>
              <surname>Baessler</surname>
            </name>
          </person-group>
          <source>Insights Imaging</source>
          <volume>11</volume>
          <fpage>91</fpage>
          <lpage>91</lpage>
          <year>2020</year>
        </element-citation>
        </ref>
      <ref id="b16">
        <element-citation publication-type="journal">
          <article-title>Radiomics Based on Adapted Diffusion Kurtosis Imaging Helps to Clarify Most Mammographic Findings Suspicious for Cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>S</given-names>
              <surname>Bickelhaupt</surname>
            </name>
            <name>
              <given-names>P F</given-names>
              <surname>Jaeger</surname>
            </name>
            <name>
              <given-names>F B</given-names>
              <surname>Laun</surname>
            </name>
            <name>
              <given-names>W</given-names>
              <surname>Lederer</surname>
            </name>
            <name>
              <given-names>H</given-names>
              <surname>Daniel</surname>
            </name>
            <name>
              <given-names>T A</given-names>
              <surname>Kuder</surname>
            </name>
          </person-group>
          <source>Radiology</source>
          <volume>287</volume>
          <fpage>761</fpage>
          <lpage>770</lpage>
          <year>2018</year>
        </element-citation>
        </ref>
      <ref id="b5">
        <element-citation publication-type="misc">
          <person-group person-group-type="author">
            <name>
              <given-names>K J</given-names>
              <surname>Van Zee</surname>
            </name>
            <name>
              <given-names>D M</given-names>
              <surname>Manasseh</surname>
            </name>
            <name>
              <given-names>J L</given-names>
              <surname>Bevilacqua</surname>
            </name>
            <name>
              <given-names>S K</given-names>
              <surname>Boolbol</surname>
            </name>
            <name>
              <given-names>J V</given-names>
              <surname>Fey</surname>
            </name>
            <name>
              <given-names>L K</given-names>
              <surname>Tan</surname>
            </name>
          </person-group>
          <source>A nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy</source>
        </element-citation>
        </ref>
      <ref id="b28">
        <element-citation publication-type="journal">
          <article-title>Positron emission tomography (PET) and magnetic resonance imaging (MRI) for the assessment of axillary lymph node metastases in early breast cancer: systematic review and economic evaluation</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>K L</given-names>
              <surname>Cooper</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Meng</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Harnan</surname>
            </name>
            <name>
              <given-names>S E</given-names>
              <surname>Ward</surname>
            </name>
            <name>
              <given-names>P</given-names>
              <surname>Fitzgerald</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Papaioannou</surname>
            </name>
          </person-group>
          <source>Health Technol Assess</source>
          <volume>15</volume>
          <year>2011</year>
        </element-citation>
        </ref>
      <ref id="b10">
        <element-citation publication-type="journal">
          <article-title>A Model to Predict the Risk of Lymph Node Metastasis in Breast Cancer Based on Clinicopathological Characteristics</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>W</given-names>
              <surname>Chen</surname>
            </name>
            <name>
              <given-names>C</given-names>
              <surname>Wang</surname>
            </name>
            <name>
              <given-names>F</given-names>
              <surname>Fu</surname>
            </name>
            <name>
              <given-names>B</given-names>
              <surname>Yang</surname>
            </name>
            <name>
              <given-names>C</given-names>
              <surname>Chen</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Sun</surname>
            </name>
          </person-group>
          <source>Cancer Manag Res</source>
          <volume>12</volume>
          <fpage>10439</fpage>
          <lpage>10447</lpage>
          <year>2020</year>
        </element-citation>
        </ref>
      <ref id="b21">
        <element-citation publication-type="journal">
          <article-title>MRI-Based Radiomics Nomogram: Prediction of Axillary Non-Sentinel Lymph Node Metastasis in Patients With Sentinel Lymph Node-Positive Breast Cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>Y</given-names>
              <surname>Qiu</surname>
            </name>
            <name>
              <given-names>X</given-names>
              <surname>Zhang</surname>
            </name>
            <name>
              <given-names>Z</given-names>
              <surname>Wu</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Wu</surname>
            </name>
            <name>
              <given-names>Z</given-names>
              <surname>Yang</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Wang</surname>
            </name>
          </person-group>
          <source>Front Oncol</source>
          <volume>12</volume>
          <fpage>811347</fpage>
          <lpage>811347</lpage>
          <year>2022</year>
        </element-citation>
        </ref>
      <ref id="b9">
        <element-citation publication-type="journal">
          <article-title>Correlation of tumor volume and surface area with lymph node status in patients with multifocal/multicentric breast carcinoma</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>A A</given-names>
              <surname>Andea</surname>
            </name>
            <name>
              <given-names>D</given-names>
              <surname>Bouwman</surname>
            </name>
            <name>
              <given-names>T</given-names>
              <surname>Wallis</surname>
            </name>
            <name>
              <given-names>D W</given-names>
              <surname>Visscher</surname>
            </name>
          </person-group>
          <source>Cancer</source>
          <volume>100</volume>
          <fpage>20</fpage>
          <lpage>27</lpage>
          <year>2004</year>
        </element-citation>
        </ref>
      <ref id="b2">
        <element-citation publication-type="journal">
          <article-title>Evaluation of different imaging modalities for axillary lymph node staging in breast cancer patients to provide a personalized and optimized therapy algorithm</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>J</given-names>
              <surname>Diessner</surname>
            </name>
            <name>
              <given-names>L</given-names>
              <surname>Anders</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Herbert</surname>
            </name>
            <name>
              <given-names>M</given-names>
              <surname>Kiesel</surname>
            </name>
            <name>
              <given-names>T</given-names>
              <surname>Bley</surname>
            </name>
            <name>
              <given-names>T</given-names>
              <surname>Schlaiss</surname>
            </name>
          </person-group>
          <source>J Cancer Res Clin Oncol</source>
          <fpage>2023</fpage>
          <lpage>2023</lpage>
        </element-citation>
        </ref>
      <ref id="b22">
        <element-citation publication-type="journal">
          <article-title>A nomogram based on radiomics signature and deeplearning signature for preoperative prediction of axillary lymph node metastasis in breast cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>D</given-names>
              <surname>Wang</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Hu</surname>
            </name>
            <name>
              <given-names>C</given-names>
              <surname>Zhan</surname>
            </name>
            <name>
              <given-names>Q</given-names>
              <surname>Zhang</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Wu</surname>
            </name>
            <name>
              <given-names>Ai T</given-names>
              <surname/>
            </name>
          </person-group>
          <source>Front Oncol</source>
          <volume>12</volume>
          <fpage>940655</fpage>
          <lpage>940655</lpage>
          <year>2022</year>
        </element-citation>
        </ref>
      <ref id="b8">
        <element-citation publication-type="journal">
          <article-title>Assessment of axillary lymph node involvement in small breast cancer: analysis of 893 cases</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>B</given-names>
              <surname>Cutuli</surname>
            </name>
            <name>
              <given-names>M</given-names>
              <surname>Velten</surname>
            </name>
            <name>
              <given-names>C</given-names>
              <surname>Martin</surname>
            </name>
          </person-group>
          <source>Clin Breast Cancer</source>
          <volume>2</volume>
          <fpage>59</fpage>
          <lpage>66</lpage>
          <year>2001</year>
        </element-citation>
        </ref>
      <ref id="b24">
        <element-citation publication-type="journal">
          <article-title>Preoperative Prediction of Axillary Lymph Node Metastasis in Breast Cancer Based on Intratumoral and Peritumoral DCE-MRI Radiomics Nomogram</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>Y</given-names>
              <surname>Liu</surname>
            </name>
            <name>
              <given-names>X</given-names>
              <surname>Li</surname>
            </name>
            <name>
              <given-names>L</given-names>
              <surname>Zhu</surname>
            </name>
            <name>
              <given-names>Z</given-names>
              <surname>Zhao</surname>
            </name>
            <name>
              <given-names>T</given-names>
              <surname>Wang</surname>
            </name>
            <name>
              <given-names>X</given-names>
              <surname>Zhang</surname>
            </name>
          </person-group>
          <source>Contrast Media Mol Imaging</source>
          <volume>2022</volume>
          <issue>18</issue>
          <fpage>6729473</fpage>
          <lpage>6729473</lpage>
        </element-citation>
        </ref>
      <ref id="b26">
        <element-citation publication-type="journal">
          <article-title>Management of the Node-Positive Axilla in Breast Cancer in 2017: Selecting the Right Option</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>M</given-names>
              <surname>Morrow</surname>
            </name>
          </person-group>
          <source>JAMA Oncol</source>
          <volume>4</volume>
          <fpage>250</fpage>
          <lpage>251</lpage>
          <year>2018</year>
        </element-citation>
        </ref>
      <ref id="b19">
        <element-citation publication-type="journal">
          <article-title>Preoperative prediction of sentinel lymph node metastasis in breast cancer based on radiomics of T2-weighted fat-suppression and diffusion-weighted MRI</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>Y</given-names>
              <surname>Dong</surname>
            </name>
            <name>
              <given-names>Q</given-names>
              <surname>Feng</surname>
            </name>
            <name>
              <given-names>W</given-names>
              <surname>Yang</surname>
            </name>
            <name>
              <given-names>Z</given-names>
              <surname>Lu</surname>
            </name>
            <name>
              <given-names>C</given-names>
              <surname>Deng</surname>
            </name>
            <name>
              <given-names>L</given-names>
              <surname>Zhang</surname>
            </name>
          </person-group>
          <source>Eur Radiol</source>
          <volume>28</volume>
          <fpage>582</fpage>
          <lpage>591</lpage>
          <year>2018</year>
        </element-citation>
        </ref>
      <ref id="b18">
        <element-citation publication-type="journal">
          <article-title>Dedicated Axillary MRI-Based Radiomics Analysis for the Prediction of Axillary Lymph Node Metastasis in Breast Cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>S</given-names>
              <surname>Samiei</surname>
            </name>
            <name>
              <given-names>Rwy</given-names>
              <surname>Granzier</surname>
            </name>
            <name>
              <given-names>A</given-names>
              <surname>Ibrahim</surname>
            </name>
            <name>
              <given-names>S</given-names>
              <surname>Primakov</surname>
            </name>
            <name>
              <given-names>Mbi</given-names>
              <surname>Lobbes</surname>
            </name>
            <name>
              <given-names>Rgh</given-names>
              <surname>Beets-Tan</surname>
            </name>
          </person-group>
          <source>Cancers (Basel)</source>
          <volume>2021</volume>
          <fpage>757</fpage>
          <lpage>757</lpage>
        </element-citation>
        </ref>
      <ref id="b0">
        <element-citation publication-type="journal">
          <article-title>Axillary lymph node status, but not tumor size, predicts locoregional recurrence and overall survival after mastectomy for breast cancer</article-title>
          <person-group person-group-type="author">
            <name>
              <given-names>S W</given-names>
              <surname>Beenken</surname>
            </name>
            <name>
              <given-names>M M</given-names>
              <surname>Urist</surname>
            </name>
            <name>
              <given-names>Y</given-names>
              <surname>Zhang</surname>
            </name>
            <name>
              <given-names>R</given-names>
              <surname>Desmond</surname>
            </name>
            <name>
              <given-names>H</given-names>
              <surname>Krontiras</surname>
            </name>
            <name>
              <given-names>H</given-names>
              <surname>Medina</surname>
            </name>
          </person-group>
          <source>Ann Surg</source>
          <volume>237</volume>
          <fpage>732</fpage>
          <lpage>739</lpage>
          <year>2003</year>
        </element-citation>
        </ref>
    </ref-list>
  </back>
</article>
