Research Summary
This study presents a deep learning framework that automatically detects routine neonatal care activities—including diaper changing, tube feeding, and patting—from bedside video recordings and combines them with physiological signals such as heart rate and oxygen saturation. The system accurately quantifies the duration, frequency, and physiological impact of caregiving activities, enabling objective assessment of neonatal stress and laying the foundation for continuous AI-assisted stress monitoring in the NICU.
Premature infants admitted to Neonatal Intensive Care Units (NICUs) experience numerous routine caregiving procedures every day, many of which contribute to cumulative physiological stress that may adversely affect neurodevelopment. Existing approaches for measuring neonatal stress rely on manual documentation using the Neonatal Infant Stressor Scale (NISS), making continuous monitoring impractical in busy clinical environments.
This study developed a multimodal deep learning system that analyzes bedside video recordings together with synchronized physiological signals to automatically detect neonatal care manipulation activities, quantify their duration and frequency, estimate immediate physiological responses, and calculate stress exposure. The framework was trained and evaluated using 289 hours of synchronized video and physiological recordings collected during 330 clinical sessions from 27 neonates across two NICUs. Using advanced computer vision models and temporal activity localization techniques, the system achieved activity quantification statistically equivalent to expert human annotation within clinically acceptable error margins while accurately identifying activities associated with significant physiological changes.
The proposed framework represents an important step toward continuous, objective, AI-powered neonatal stress monitoring capable of supporting clinicians with standardized documentation, automated Neonatal Infant Stressor Scale (NISS) estimation, and real-time decision support in intensive care environments.