What it does

Sustain is a mobile app that uses AI to detect when a musician is practicing, and provides feedback on their practice sessions. The app is designed to help musicians of all levels improve their practice habits and achieve their musical goals.

Features

• Watches for instrument presence (AI object detection).
• Listens for instrument sound (AI sound classification).
• Rewards practice points while the instrument is seen and heard.
• Displays encouragements while you practice.
• Logs your practice sessions.

Future Features

• Rewards for streaks (consecutive days practiced) and cumulative time practiced.
• Additional spotter faces.
• Leaderboards or leagues.
• Practice quality feedback (intonation, tone quality).

Usage

1. Download the app from the App Store or Google Play.
2. Open the app and grant camera and microphone permissions.
3. Position your device to see your instrument and hands (a tripod is recommended).
4. Select your instrument and practice duration in the Settings menu.
5. Click the start button to begin your practice session (progress is measured only while you actively practice).

Technical Details

1. Ultralytics Platform to annotate over 600 images (both natural and synthetic) of various instruments and playing techniques.
2. Ultralytics YOLO26 instance segmentation base model, trained on our 600 images (https://docs.ultralytics.com/models/yolo26).
3. YAMNet audio classification model (https://www.kaggle.com/models/google/yamnet).
4. Flutter development framework (https://flutter.dev/) in Visual Studio Code.
5. App bundle build and deployment through GitHub, Apple XCode, and Android Studio.