P061 - CLINICAL EVALUATION OF SPICY-AI IN A RAPID ACCESS CHEST PAIN CLINIC: A COMPARATIVE VALIDATION STUDY OF A NOVEL, LOCALLY DEPLOYED AI MODEL VERSUS PHYSICIANS AND GPT-4O.

Background: Rapid Access Chest Pain Clinics (RACPCs) streamline diagnosis and cardiovascular risk optimization. SPICY-AI (Smart Platform for Integrated Chest Pain Evaluation Yielded by Artificial Intelligence) is a novel, locally deployed retrieval-augmented generation (RAG) large language model designed to generate evidence-based clinical plans. This study evaluated the clinical accuracy, documentation quality, and safety of SPICY-AI compared to physician-generated plans and GPT-4o in real-world RACPC cases.

METHODS AND RESULTS: This single-centre study analyzed 30 consecutive RACPC attendances from September to October 2024. A panel of cardiology physicians scored the clinical accuracy and guideline adherence (diagnostics and management) and documentation quality (completeness and clarity) on 5-point Likert scales for anonymized plans generated by physicians, SPICY-AI, and GPT-4o (a cloud-based large language model; n=90). Hallucinations and errors were quantified as proportions. Statistical comparisons used Mann-Whitney U and Chi-square tests.

SPICY-AI significantly outperformed physicians in documentation completeness (p=0.0004) and clarity (p=0.0002). GPT-4o also surpassed physicians (completeness: p=0.0003; clarity: p=0.0003). SPICY-AI and GPT-4o performed equivalently across all domains (all p>0.05). GPT-4o significantly exceeded physicians in the clinical accuracy of management plans (p=0.016), with SPICY-AI showing borderline significance (p=0.018). Diagnostic accuracy and error rates were comparable among groups, with no significant differences.

Conclusion: SPICY-AI matched physicians in diagnostic work-up accuracy but outperformed them in cardiovascular risk optimization planning, delivering clearer, more comprehensive, and guideline-adherent plans. Local deployment addresses data privacy concerns associated with cloud-based models such as GPT-4o, highlighting SPICY-AI’s potential to safely support protocolized clinical assessments in RACPCs.