Clinical Validation of an Artificial Intelligence System for the Interpretation of Laboratory Test Results Combined With a Structured Medical History

The urgency category assigned by the software is compared with the urgency category assigned by an expert physician serving as the reference standard. The urgency categories, ordered from highest to lowest, are: immediate medical contact, urgent medical contact, routine medical contact, and no need for medical contact. A participant is counted in the numerator when the category assigned by the software is the same as or higher than the category assigned by the expert physician. The measure is the percentage of participants counted in the numerator. Range: 0 to 100 percent; higher values indicate better performance. The measure is assessed separately in each study phase and data from the two phases are not combined. Each comparison is based on data locked at the time of enrollment; the expert assessment is performed retrospectively, in batches, on a locked dataset, and the calendar timing of the batch review does not affect the measured quantity.

The urgency category assigned by the software is compared with the urgency category assigned by an expert physician serving as the reference standard. The urgency categories, ordered from highest to lowest, are: immediate medical contact, urgent medical contact, routine medical contact, and no need for medical contact. A participant is counted in the numerator when the category assigned by the software is identical to the category assigned by the expert physician. The measure is the percentage of participants counted in the numerator. Range: 0 to 100 percent; higher values indicate better performance. The measure is assessed separately in each study phase and data from the two phases are not combined. Each comparison is based on data locked at the time of enrollment; the expert assessment is performed retrospectively, in batches, on a locked dataset, and the calendar timing of the batch review does not affect the measured quantity.

Proportion of participants assigned by the expert physician to the immediate medical contact category who were assigned to the same category by the software. Range: 0 to 100 percent; higher values indicate better performance. Reported with a 95 percent confidence interval and the number of events. No formal hypothesis test is performed, because the expected denominator at the planned sample size is not sufficient for confirmatory testing. Assessed separately in both study phases.

Proportion of participants assigned by the expert physician to the urgent medical contact category who were assigned to the same category by the software. Range: 0 to 100 percent; higher values indicate better performance. Reported with a 95 percent confidence interval and the number of events. No formal hypothesis test is performed, because the expected denominator at the planned sample size is not sufficient for confirmatory testing. Assessed separately in both study phases.

Agreement between the urgency category assigned by the software and by the expert physician, expressed as Cohen's kappa with quadratic weights applied to the four ordered urgency categories. Range: -1 to 1; higher values indicate better agreement. Reported with a 95 percent confidence interval. Supplementary to the co-primary measure of accuracy. Assessed separately in both study phases.

Concordance between the medical specialty recommended by the software and by the expert physician, evaluated at the level of a single laboratory result. Reported as precision and macro-averaged recall. Range: 0 to 1; higher values indicate better performance. Assessed separately in both study phases.

Concordance between the additional laboratory tests recommended by the software and by the expert physician, evaluated at the level of a single laboratory result. Reported as precision and macro-averaged recall. Range: 0 to 1; higher values indicate better performance. Assessed separately in both study phases.

Concordance of three sources of assessment - the software, the attending physician and large language models - with the same expert reference assessment, using the measures defined above. Assessments are compared retrospectively on a locked dataset. Reported descriptively for each source; no ranking test is performed.

Proportion of participants who, in a study-specific questionnaire completed after release of the interpretation report, indicate that the content of the report was comprehensible. Range: 0 to 100 percent; higher values indicate better performance. Assessed separately in both study phases.

Proportion of automated medical history records for which the attending physician reports no objections regarding completeness or relevance, recorded in a study-specific form. Range: 0 to 100 percent; higher values indicate better performance. Assessed separately in both study phases.