Medical Large Language Model-Assisted Diagnosis and Treatment in Primary Care Chronic Disease Management

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale on several aspects: accuracy of condition assessment, reasonableness of test recommendations, reasonableness of treatment plans, appropriateness of management choices, scientific nature of patient education content, feasibility at the primary care level, overall clinical quality. Experts were instructed to rank the seven dimensions from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale on accuracy of condition assessment from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the reasonableness of test recommendations. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the reasonableness of treatment plans. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the ppropriateness of management choices. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the scientific nature of patient education content. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the feasibility at the primary care level. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the overall clinical quality. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. If there is no risk advice, the expert selects 'No'; otherwise, the expert selects 'Yes' and specifically lists the content of the risk advice.

A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. If there is no omission of diagnostic and treatment information, the expert selects "No"; otherwise, the expert selects "Yes" and specifically lists the omitted diagnostic and treatment content.

After primary care physicians complete a case assessment assisted by medical LLM , they will fill out an evaluation of the medical LLM usage, including whether the medical LLM helped in disease assessment, assisted in formulating examination plans, aided in developing treatment plans, improved chronic disease management capabilities, supported chronic disease education, assisted in management approach selection, enhanced confidence, saved time, whether there were hallucinations or omissions, whether it provided risky suggestions, and any additional comments on the use of the medical LLM.