SeeMe: Using Automated Facial Tracking to Detect Voluntary Behavior in Brain Injury

• Study Rationale and Scientific Gap Standard clinical assessments for Traumatic Brain Injury (TBI), such as the Glasgow Coma Scale (GCS) and Coma Recovery Scale-Revised (CRS-R), are limited by human subjectivity, examiner bias, and low spatiotemporal resolution. Recent multicenter studies indicate that up to 25% of patients who appear "unresponsive" at the bedside may exhibit "Cognitive-Motor Dissociation" (CMD)-a state of covert awareness where the brain intends to move, but motor output is too microscopic for the naked eye to detect. Misclassifying these patients as unresponsive negatively impacts clinical outcomes due to the potential withholding of rehabilitative resources. This study validates SeeMe, a multimodal platform designed to bridge this diagnostic gap by objectively quantifying microscopic behavior in real-time.
• The SeeMe Technological Framework SeeMe is an innovative computer-vision sensing platform that utilizes vector analysis, cross-correlation, and deep neural networks (DNNs) to track individual facial pores and hand landmarks.

Sub-millimeter Precision: The system offers high spatial precision (0.5 mm) and high temporal resolution (0.03s), allowing it to capture "micro-expressions" and low-amplitude motor responses (e.g., vertical eye movements or subtle mouth twitches) that escape clinical detection.

Advanced Classification: The investigators utilize a bidirectional long short-term memory (LSTM) network to classify heatmaps of facial movement. This ensures that detected responses are specific to the auditory command provided (e.g., "Show me a smile" vs. "Open your eyes") rather than generalized arousal or non-specific reflexive movements.

• Multimodal Brain-Behavior Synchronization The primary objective of the R61 phase is the seamless integration of SeeMe with electrophysiological activity (EEG/ECoG). By time-locking microscopic motor initiation with neural oscillations, the investigators aim to identify objective biomarkers of recovery.

Beta-Band Signatures: The investigators focus on Beta-band Event-Related Desynchronization (ERD) and Synchronization (ERS) as signatures of motor planning and execution.

One-to-One Mapping: This multimodal approach provides a robust computational framework to synchronize brain activity with behavior, allowing us to map the "bigger picture" of recovery during the critical transition from coma to consciousness.

• Study Design and Methodology

This prospective validation study will utilize three distinct cohorts (Total N = 80-100):

Cohort 1: Healthy Controls (n=10): Establishes the "ground truth" for robust, voluntary motor movements and baseline algorithm performance.

Cohort 2: Sedated/Anesthetized Controls (n=10): Patients undergoing spine surgery under general anesthesia/pharmacological paralysis provide a "zero-movement" baseline to quantify the algorithm's noise floor and establish specificity.

Cohort 3: Recovering TBI Patients (n=60-80): The primary clinical cohort.

Procedures: Participants listen to simple auditory commands (e.g., "Stick out your tongue," "Open your eyes," "Close your hands") alongside a control command ("Today is a sunny day") to distinguish auditory-evoked movements from true command following.

Comparison: SeeMe data will be collected daily and compared against blinded CRS-R assessments to determine the "detection lead time"-the number of days SeeMe identifies a response before the clinical team notes command following.

• Clinical and Neuroethical Oversight

Acknowledging the vulnerability of the TBI population, the study incorporates two novel neuroethical safeguards:

Patient Advocate: Modeled on pioneering work in organ donation, a dedicated liaison will be appointed to act as a bridge between the research team and families, ensuring participant safety and well-being remain the central focus.

TBI-Affected Family Council: A consultative council of family members who have experienced TBI will review findings and provide guidance on the ethical communication of results to families.

• Future Directions The data gathered during this R61 phase-specifically the synchronization of SeeMe and EEG-will define the physiological triggers and milestones for Phase R33. In the subsequent phase, the SeeMe-EEG platform will be integrated with Vagus Nerve Stimulation (VNS). This closed-loop system will test the hypothesis that precisely-timed VNS pulses, paired with SeeMe-detected motor intentions, can facilitate the reorganization of motor circuits and accelerate the recovery of goal-directed behavior after TBI.

Group 1: Traumatic Brain Injury (TBI) Cohort

Inclusion criteria

• Adults (22+) with a history of acute traumatic brain injury
• Documented loss of consciousness with a Glasgow Coma Scale (GCS) score less than or equal to 8 upon hospitalization
• Clinically stable as determined by the primary neurosurgery or ICU team
• Intact auditory pathways as confirmed by BAERs
• Family consent for study participation

Exclusion criteria

• Hearing Impairment confirmed via absence of Brainstem Auditory Evoked Responses (BAERs) that would prevent the patient from hearing the auditory commands
• No legal authorized representative (LAR) available to provide informed consent for the patients in a comatose state
• Any other medical condition that, in the judgment of the investigator, makes participation in the study unsafe.
• Pregnant women
• Any previous history of traumatic brain injury
• Any neurodegenerative disease such as dementia

Group 2: Healthy Control Cohort

Inclusion criteria

• Adults 22+ with no history of neurological or psychiatric disorders
• Normal baseline neurological examination
• Intact auditory pathways
• Ability to provide informed consent
• Ability to follow simple auditory commands in English

Exclusion criteria

• Hearing Impairment that would prevent the participant from hearing the auditory commands
• Any previous history of severe traumatic brain injury (TBI)
• Any neurodegenerative disease (e.g., dementia)
• Any motor impairment (e.g., facial palsy, carpal tunnel syndrome) that would interfere with facial or hand movement tracking
• Any other medical condition that, in the judgment of the investigator, makes participation in the study unsafe
• Pregnant women

Group 3: Sedated/Anesthetized Cohort

Inclusion criteria

• Adults (22+) undergoing elective spine surgery
• Requirement of general anesthesia and pharmacological paralysis (neuromuscular blockade) as part of the standard surgical procedure
• Clinically stable for study procedures as determined by the anesthesia and surgical teams.
• Intact auditory pathways
• Ability to provide informed pre-operative consent

Exclusion criteria

• Hearing Impairment that would prevent the patient from hearing the auditory commands
• Any previous history of severe traumatic brain injury
• Any neurodegenerative disease such as dementia
• Significant baseline facial or hand motor deficits prior to the administration of anesthesia
• Any other medical condition that, in the judgment of the investigator, makes participation in the study unsafe
• Pregnant women