Comparative Diagnostic Accuracy of Large Language Models for STEMI Classification An Exploratory Study Using De-Identified Emergency Department Cases
Article Sidebar
Main Article Content
Abstract
Background: Early and accurate identification of ST-Elevation Myocardial Infarction (STEMI) in the prehospital setting would reduce morbidity and mortality (Rao et al. 2025). Artificial Intelligence (AI) tools may assist emergency medical personnel by providing rapid electrocardiogram (ECG) interpretation and differential diagnosis (Chen et al. 2022; Nallamothu et al., 2015).
Objective: To evaluate the diagnostic accuracy of ChatGPT-4.0, Gemini 2.5 Pro, and a hybrid model combining ECG-GPT + ChatGPT-4.0 in classifying STEMI versus non-STEMI cases based on ECG and clinical data inputs.
Methods: Fifty-six consecutive de-identified cases (28 STEMI, 28 non-STEMI) from Staten Island University Hospital EMR (1/2025-6/2025) were analyzed. Each case included demographics, vital signs, chief complaints, and a representative 12-lead ECG. The three AI models were independently asked to classify each case as STEMI or not STEMI.
Results: For the STEMI cohort (n=28), detection rates were: ChatGPT-4.0 21.4%, Gemini 2.5 Pro 67.9%, and ECG-GPT + ChatGPT hybrid 71.4%. For the non-STEMI cohort (n=28), correct classification rates were: ChatGPT-4.0 92.9%, Gemini 2.5 Pro 28.6%, and ECG-GPT + ChatGPT hybrid: 96.4%. The ECG-GPT + ChatGPT hybrid model demonstrated the highest diagnostic accuracy at 83.9%.
Conclusion: While ChatGPT-4.0 showed high specificity, it lacked sensitivity for STEMI detection. Gemini 2.5 Pro improved sensitivity but yielded higher false positives. The hybrid ECG-GPT + ChatGPT model outperformed both standalone models, suggesting that multimodal AI integration may enhance triage accuracy for prehospital care. Despite the small population size and limitations associated with this, these results propose a viable means to hypothesize that AI may have potential in assisting responders with early STEMI identification. Future improvements in model precision could support the implementation of AI-based triage systems in emergency response settings, reducing time to treatment and improving outcomes by aiding direct cardiac catheterization lab transfers without unnecessary delays (Garvey et al., 2012; Squire et al., 2014).
Article Details
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
Publishing in IJOP allows authors to keep their copyright while giving IJOP unrestricted copyright permissions. Articles published in IJOP use Creative Common Attribution 4.0 International (CC BY-ND 4.0) licensing. This license requires that re-users give credit to the creator. It allows re-users to copy and distribute the material in any medium or format in unadapted form only, even for commercial purposes. Additional terms apply and can be accessed here.
Publishing in IJOP also allows authors to have contracts for non-exclusive distribution of the Journal's published version of the article, such as posting to an institutional repository or publication in a book, on the condition that the original publication in the original layout format in IJOP is retained and acknowledged.
We permit and encourage authors to post the articles they published in IJOP on their affiliated websites. This helps share the information, encourages citation in other works, and promotes scholarly discourse in the spirit of open access.
References
Chen, K. W., Wang, Y. C., Liu, M. H., Tsai, B. Y., Wu, M. Y., Hsieh, P. H., Wei, J. T., Shih, E. S. C., Shiao, Y. T., Hwang, M. J., Wu, Y. L., Hsu, K. C., & Chang, K. C. (2022). Artificial intelligence-assisted remote detection of ST-elevation myocardial infarction using a mini-12-lead electrocardiogram device in prehospital ambulance care. Frontiers in Cardiovascular Medicine, 9, Article 1001982. https://doi.org/10.3389/fcvm.2022.1001982
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., et al. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, Article e078378. https://doi.org/10.1136/bmj-2023-078378
Garvey, J. L., Monk, L., Granger, C. B., Studnek, J. R., Roettig, M. L., Corbett, C. C., & Jollis, J. G. (2012). Rates of cardiac catheterization cancelation for ST-segment elevation myocardial infarction after activation by emergency medical services or emergency physicians: Results from the North Carolina Catheterization Laboratory Activation Registry. Circulation, 125(2), 308–313. https://doi.org/10.1161/CIRCULATIONAHA.110.007039
Gregory, P., Lodge, S., Kilner, T., & Paget, S. (2019). Accuracy of ECG chest electrode placements by paramedics: An observational study. British Paramedic Journal, 4(3), 51–52. https://doi.org/10.29045/14784726.2019.12.4.3.51
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019). Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1), 65-69. https://doi.org/10.1038/s41591-018-0268-3
Khunte, A., Sangha, V., Oikonomou, E. K., Dhingra, L. S., et al. (2026). Artificial intelligence based automated interpretation of images of electrocardiograms: Development and multinational validation of ECG-GPT. European Heart Journal—Digital Health, 7(3). https://doi.org/10.1093/ehjdh/ztag031
Lai, J. H. Y., Lui, C. T., Chan, T. W. T., Wong, B. C. P., Tsui, M. S. H., Wan, B. K. A., & Mok, K. L. (2024). Diagnostic accuracy of a prehospital electrocardiogram rule-based algorithm for ST elevation myocardial infarction: Results from a population-wide project. Hong Kong Medical Journal, 30(4), 271–280. https://doi.org/10.12809/hkmj2310827
Larson, D. M., Menssen, K. M., Sharkey, S. W., Duval, S., Schwartz, R. S., Harris, J., Meland, J. T., Unger, B. T., & Henry, T. D. (2007). “False-positive” cardiac catheterization laboratory activation among patients with suspected ST-segment elevation myocardial infarction. JAMA, 298(23), 2754–2760. https://doi.org/10.1001/jama.298.23.2754
Mencl, F., Wilber, S., Frey, J., Zalewski, J., Maiers, J. F., & Bhalla, M. C. (2013). Paramedic ability to recognize ST-segment elevation myocardial infarction on prehospital electrocardiograms. Prehospital Emergency Care, 17(2), 203–210. https://doi.org/10.3109/10903127.2012.755585
Nallamothu, B. K., Normand, S. L., Wang, Y., Hofer, T. P., Brush, J. E., Jr., Messenger, J. C., Bradley, E. H., Rumsfeld, J. S., & Krumholz, H. M. (2015). Relation between door-to-balloon times and mortality after primary percutaneous coronary intervention over time: A retrospective study. Lancet, 385(9973), 1114–1122. https://doi.org/10.1016/S0140-6736(14)61932-2
Pilbery, R., Teare, M. D., Goodacre, S., & Morris, F. (2016). The recognition of STEMI by paramedics and the effect of computer interpretation: A randomised crossover feasibility study. Emergency Medicine Journal, 33(7), 471–476. https://doi.org/10.1136/emermed-2015204988
Rao, S. V., O'Donoghue, M. L., Ruel, M., Rab, T., Tamis-Holland, J. E., Alexander, J. H., Baber, U., Baker, H., Cohen, M. G., Cruz-Ruiz, M., Davis, L. L., de Lemos, J. A., DeWald, T. A., Elgendy, I. Y., Feldman, D. N., Goyal, A., Isiadinso, I., Menon, V., Morrow, D. A., Mukherjee, D., … Williams, M. S. (2025). 2025 ACC/AHA/ACEP/NAEMSP/SCAI guideline for the management of patients with acute coronary syndromes: A report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation, 151(13), e771–e862. https://doi.org/10.1161/CIR.0000000000001309
Squire, B. T., Tamayo-Sarver, J. H., Rashi, P., Koenig, W., & Niemann, J. T. (2014). Effect of prehospital cardiac catheterization lab activation on door-to-balloon time, mortality, and false positive activation. Prehospital Emergency Care, 18(1), 1–8. https://doi.org/10.3109/10903127.2013.836263