Premature infants admitted to Neonatal Intensive Care Units (NICUs) experience numerous routine caregiving procedures every day, many of which contribute to cumulative physiological stress that may adversely affect neurodevelopment. Existing approaches for measuring neonatal stress rely on manual documentation using the Neonatal Infant Stressor Scale (NISS), making continuous monitoring impractical in busy clinical environments.
This study developed a multimodal deep learning system that analyzes bedside video recordings together with synchronized physiological signals to automatically detect neonatal care manipulation activities, quantify their duration and frequency, estimate immediate physiological responses, and calculate stress exposure. The framework was trained and evaluated using 289 hours of synchronized video and physiological recordings collected during 330 clinical sessions from 27 neonates across two NICUs. Using advanced computer vision models and temporal activity localization techniques, the system achieved activity quantification statistically equivalent to expert human annotation within clinically acceptable error margins while accurately identifying activities associated with significant physiological changes.
The proposed framework represents an important step toward continuous, objective, AI-powered neonatal stress monitoring capable of supporting clinicians with standardized documentation, automated Neonatal Infant Stressor Scale (NISS) estimation, and real-time decision support in intensive care environments.