Specialist Review of Harm Ratings for Clinical Responses Acronym: NOHARM

The purpose of this study is to determine how closely potential-harm ratings assigned using the NOHARM rubric agree with independent ratings from specialty-matched physicians. The study uses reconstructed clinical consultation cases and previously generated responses written by artificial intelligence systems or physicians. The materials contain no patient identifiers or protected health information. Board-certified attending physicians from 10 medical specialties will review assigned case-response pairs through a secure website. For each response, they will indicate whether it contains an error, describe the error when applicable, and rate the greatest potential patient harm as none, mild, moderate, or severe. Reviewers will not be shown the response source, existing rubric rating, or other reviewers' ratings. Two specialists will initially review each response; a third may review it if the first two ratings differ. The study includes a development stage, possible rubric revision, and a separate validation stage.

Not yet recruiting

Trial opening soon.

Get Notified

Key information

Conditions

Patient Safety Medical Errors

Age range

18 year and older

Sex eligibility

All sexes

Study type

Observational

Who can participate

Healthy volunteers accepted: Yes

Only the study team can determine whether someone qualifies for participation.

Inclusion criteria

Licensed or registered physician in good standing
Attending- or consultant-level physician actively practicing in allergy and immunology, cardiology, dermatology, endocrinology, gastroenterology, hematology, infectious disease, nephrology, neurology, or pulmonology
Practicing in the United States, Canada, United Kingdom, or Australia
Assigned clinical materials match the physician's specialty
Able to read and respond in English
Access to a computer with a stable internet connection
Willing and able to complete at least one remote specialist-rating engagement of approximately three hours

Exclusion criteria

Does not meet the professional, career-level, specialty, geographic, or credential-verification requirements above
Prior involvement with assigned cases, responses, or rubric information that could compromise independent rating
Unable to complete the remote rating activities in English
Unable or unwilling to follow the independent-rating procedures
For validation-stage assignments, participation in the rubric-revision process

Treatment and study plan

Primary outcomes

Inverse-Probability-Weighted Confirmation of Rubric-Severe Classifications

Time frame: At completion of the development and validation stages, up to 2 months

The inverse-probability-weighted proportion of responses classified as severe by the NOHARM rubric that are also classified as severe by specialist consensus: P(specialist severe | rubric severe). This measure will be calculated separately for the development and validation stages. Values range from 0 to 1; higher values indicate greater confirmation. The prespecified threshold is 0.70.
Inverse-Probability-Weighted Capture of Specialist-Severe Classifications

Time frame: At completion of the development and validation stages, up to 2 months

The inverse-probability-weighted proportion of responses classified as severe by specialist consensus that are also classified as severe by the NOHARM rubric: P(rubric severe | specialist severe). This measure will be calculated separately for the development and validation stages. Values range from 0 to 1; higher values indicate greater capture. The prespecified threshold is 0.70. Both co-primary measures must meet this threshold.

Study contacts

Contact information is provided by the study sponsor or research team.

Jonathan H Chen, MD, PHD

CONTACT

[email protected]

650-725-3655

Sponsors and collaborators

Lead sponsor

Stanford University

Other

Registry information

Official study title

Specialist Physician Validation of the NOHARM Harm-Rating Rubric for Clinical Responses

Important dates

Study start: 2026
Primary completion: 2026
Study completion: 2026
First posted: Sep 16, 2026
Registry last updated: Sep 16, 2026

OpenTrials presents study information sourced from ClinicalTrials.gov. The official registry record should be consulted for the latest information.

View the official ClinicalTrials.gov record (opens in a new tab)

This listing is for discovery and informational purposes only. It is not medical advice, does not guarantee that a study is recruiting, and does not determine eligibility. Contact the study team and a qualified healthcare professional when considering participation.

Published trials that share one or more normalized conditions with this study.

Completed