Evaluating the effect of heart and respiratory rate measurement errors on the ability to predict the outcome of high flow nasal cannula therapy: a multicentre study

Background: The respiratory rate oxygenation (ROX) index, and machine learning (ML) models, are promising approaches to help clinicians identify earlier those patients at risk of failing high flow nasal cannula (HFNC) therapy. Respiratory rate (RR) and heart rate (HR) are key inputs to these models, but their measurement in a hospital environment may be subject to significant errors. The effect of these errors on the accuracy of HFNC outcome predictions is currently unknown. Methods: We evaluated the capability of a recently-proposed ML model called Tabular Prior-data Fitted Network (TabPFN), a range of standard ML models, and the ROX index and its variants, to predict the outcome of HFNC therapy using measurements made within the first 2 hours of treatment in patients with acute hypoxemic respiratory failure. 596 AHRF patients receiving HFNC (456 successes, 140 failures) from the RENOVATE trial in Brazil were used for model training. External validation was performed on a dataset on 241 AHRF patients (156 successes, 85 failures) from Italy and the US. During training and testing, we replicated RR and HR measurement errors that were consistent with those recorded in previously published studies employing 30-second and 15-second manual counting time-windows, respectively, and employed bootstrapping and Monte Carlo simulation to evaluate their effects on the accuracy of outcome predictions. Results: The TabPFN model was more affected by the RR and HR measurement errors, but still provided more accurate predictions of HFNC outcome (Mean [95% CI] Accuracy 0.79 [0.73-0.84], AUC 0.86 [0.82-0.89] in external validation) than the ROX index and its variants (Accuracy 0.71 [0.68-0.75], AUC 0.78 [0.75-0.80]). Augmenting patient datasets with arterial blood gas measurements further improved the performance and robustness of the TabPFN model, but not the ROX index. Conclusions: In this multi-centre study, the recently introduced TabPFN ML model outperformed currently available methods for predicting the outcome of HFNC therapy even when realistic levels of measurement errors were included in the clinical data on RR and HR. The predictive performance of this ML model can be further improved by minimizing measurement errors using more advanced monitoring, and/or by additionally using arterial blood gas measurements.