Large Language Model for Understanding and Monitoring Elderly Neurocognition

• Background and Rationale Dementia diagnosis relies on clinical history, cognitive assessments, and collateral information from caregivers. However, obtaining structured and reliable collateral histories is challenging due to caregiver burden, recall bias, and time constraints in memory clinics. Large language models (LLMs) have the potential to improve this process by guiding structured history-taking, extracting clinically relevant details, and standardising information gathering.

LUMEN (Large Language Model for Understanding and Monitoring Elderly Neurocognition) is an AI-powered conversational tool designed to assist caregivers in providing structured collateral histories before clinical dementia assessments. Seed-funded by the Royal College of Psychiatrists, the Alzheimer's Research UK Northern Network and the Newcastle Biomedical Research Network, LUMEN was developed to enhance diagnostic accuracy, reduce caregiver burden, and improve workflow efficiency in memory clinics.

This study will evaluate the feasibility, usability, and acceptability of LUMEN in real-world clinical settings through a nested qualitative feasibility study. We will recruit caregivers and clinicians to interact with LUMEN and assess its effectiveness in gathering collateral histories. The study will also incorporate co-production workshops with community groups to explore issues around language, cultural inclusivity, and accessibility. Findings will inform future refinements and a larger validation study.

• Research Objectives

This study aims to:

• Evaluate LUMEN's usability -Measure caregiver and clinician experiences using validated scales (System Usability Scale [SUS] and NASA Task Load Index [NASA-TLX]).
• Explore acceptability and implementation barriers and facilitators-Conduct qualitative interviews and coproduction workshops to refine LUMEN's design, ensuring accessibility across diverse user groups.
• Compare LUMEN's outputs to clinician assessments-Investigate the alignment between LUMEN-generated reports and clinician-elicited histories using Cohen's Kappa for inter-rater reliability.

• Background Work Undertaken LUMEN was co-developed with clinicians and stakeholders using a Modified Delphi process involving 150 clinicians, primarily old age psychiatrists. This produced 2,600 questions mapped to validated dementia tools, ensuring clinical relevance. Early technical testing evaluated open-source LLMs (e.g., Mistral 7B, LLaMA2 7B) using simulated clinical vignettes. ROC curve analysis demonstrated an AUC of 0.89 for distinguishing dementia-related narratives, and Cohen's Kappa 0.82 indicated high inter-rater reliability between LUMEN's outputs and clinician ratings. Early PPIE (Patient and Public Involvement and Engagement) feedback highlighted the need for simpler language, cultural sensitivity, and improved user experience. These insights will be further refined through the proposed co-production workshops.

• Study Design and Methodology This study will employ a mixed-methods approach, combining quantitative usability testing with qualitative interviews to evaluate LUMEN's feasibility and acceptability.

4.1. Participants and Recruitment

• Carers (n=20-30 dyads): Recruited via Northumbria NHS Memory Clinics. Eligible carers will be adults (≥18 years) who know the patient well, have basic English literacy, and can provide informed consent.
• Patients (n=20-30): Individuals (≥65 years) attending a memory clinic for cognitive assessment.
• Clinicians (n=8-10): Specialists in dementia care (neurologists, psychiatrists, geriatricians, advanced nurse practitioners) with ≥2 years of experience.

4.2. Study Procedures

• Baseline Data Collection:

• Demographics (age, gender, ethnicity, socioeconomic status).
• Clinical diagnosis (where available).
• Cognitive test scores (MoCA/ACE).
• LUMEN Interaction:

• Carers will use LUMEN on a laptop or tablet to provide structured collateral histories (~20-30 min).
• Clinicians will review LUMEN-generated histories to assess completeness and clinical utility.
• Usability and Cognitive Load Assessment:

• SUS (System Usability Scale): 10-item Likert-scale questionnaire assessing ease of use (score ≥70 = good usability).
• NASA-TLX (Task Load Index): Evaluates perceived cognitive workload (scores 0-29 = low workload).
• Qualitative Evaluation:

• Semi-structured interviews (n=10 carers, n=4-5 clinicians).
• Co-production workshops with community groups to explore language, cultural representation, and accessibility.
• Exploratory Analysis:

• Inter-rater reliability (Cohen's Kappa): Agreement between LUMEN outputs and clinician assessments.
• Diagnostic accuracy (AUROC curve analysis): Comparison of LUMEN's assessments with clinical diagnoses.

• Milestones and Timescales

The study will run for 12 months (April 2025 - April 2026), with key phases as follows:

Phase Month Activities Preparation & Ethics 1-3 Final IRAS approval, NHS R&D approvals, team setup Recruitment & Data Collection 4-9 Patient-carer dyad recruitment, clinician recruitment, LUMEN testing Usability Workshops 6-8 Co-production workshops, qualitative interviews Data Analysis & Refinement 9-10 Thematic analysis, SUS/NASA-TLX scoring, LLM refinements Final Reporting & Next Steps 11-12 Dissemination, grant preparation for large-scale validation study 6. Expected Impact This study will generate critical feasibility data to inform a larger clinical validation study and subsequent NHS integration. Expected outcomes include:

• A refined, co-designed LUMEN prototype with improved usability and accessibility.
• Quantitative evidence on usability and cognitive workload, supporting future funding applications.
• Preliminary insights into AI-assisted collateral history-taking, laying the foundation for regulatory approval as a Software as a Medical Device (SaMD).
• Potential for early NHS adoption under the MHRA framework, streamlining dementia diagnostics.

Carers of patients attending Memory Clinic appointments; Aged 18 or over; Both patients and cares can provide informed consent; Basic English literacy

Clinicians must have at least 2 years of specialist dementia care experience and be personally responsible for dementia diagnosis; Must be provide informed consent