PC Application
VisioTherapy was first implemented for PCs in order to verify the overall concept and allow detailed visualization of individual features, home and model exercise alignment, and detailed error analysis. The PC interface is used for continuous app development. Verified system prototypes are then translated into a mobile application that offers significantly better user experience and manipulation during exercise recording.
Web Interface
The VisioTherapy web interface is offered for therapists, who may need to view patient exercising records in more detail on a large screen to study reported problems. Furthermore, the web interface offers more advanced analysis of patients' progress and comparison between individual patients.
Technology Highlights
VisioTherapy is a unique software tool that provides personalized feedback on patient's rehabilitation exercising by comparing the patient's performance to their own reference model. In contrast to existing solutions based on multiple synchronized RGB cameras or invasive wearable sensors, VisioTherapy requires only a single camera stream (from mobile or web camera) to analyze a patient's movement. Our solution integrates existing technologies with new algorithms that enable patient monitoring in real-time, including pose estimation, segmentation, exercise evaluation and feedback.
To construct personalized exercise models, the application learns from therapist-guided exercise sessions. At the therapist's office, the patient's exercise is recorded as a video, from which a skeleton sequence is extracted and segmented into exercise repetitions. The correctly or incorrectly performed repetitions serve as references that are used to automatically personalize a set of extracted features. Subsequently, at-home exercises are similarly processed and evaluated against the personalized template. Patients get intuitive real-time feedback in mirrored video and a comprehensive summary of the entire session.
Extraction of Skeleton Data
Human motion data is extracted from an RGB mobile phone camera stream in the form of simplified skeletons, which provide coordinates of selected keypoints on the human body. The skeletons are estimated using the MediaPipe Pose tool that employs a fast mobile-compatible model with high accuracy, enabling the extraction of both 2D and 3D skeleton coordinates.
Segmentation of Exercise Repetitions
To judge the correctness of individual exercise repetitions, we need to detect their start and end. Exercise repetitions show periodic behavior in skeleton features like keypoint positions, distances between joints, and changes in bone angles. We analyze these periodicities by classifying individual frames into exercise phases using specific thresholds on handcrafted skeleton features.
Personalized Exercise Monitoring
VisioTherapy enables personalized exercise evaluation by focusing on patient-specific features. In collaboration with therapists, we define customized kinematic features – such as joint angles or relative joint positions – that capture key characteristics of each exercise. Default features validate the basic structure of the exercise (e.g., correct body posture), whereas personalized features guard exercise correctness using learned personalized thresholds and weights.
Feedback Delivery
VisioTherapy provide both real-time feedback and a comprehensive summary of a whole exercising session. The immediate feedback is computed based on default features, helping the patient correct their performance in real time. At the end of the exercise session, VisioTherapy provides an overall exercise score and a detailed breakdown identifying the most problematic body parts and movements.
The development of VisioTherapy software was co-financed from the state budget by the Technology agency of the Czech Republic under the TREND Programme.