solfege.ai ✋ 🎹
a project by Ryan Kemmer and Josh Stovall
About
We have developed a system that allows users to 'play' solfege hand signs as a MIDI instrument. Since vocalists cannot actually see the notes on their instrument, hand signs have been developed as a teaching aid in school choirs to teach music theory concepts.
Our system uses machine learning to detect these hand signs and convert them into MIDI instrument input, providing immediate auditory feedback to students. Previously, a student learning solfege would have to sing the note or play another instrument to 'hear' their hand signs. This technology has practical applications in both music education and creation.
The system utilizes TensorFlow for machine learning, MediaPipe for hand detection, and JZZ.js for MIDI conversion.
Libraries
This project uses TensorFlow, MediaPipe, and JZZ.js.
Setup
cd solfege
php -S 0.0.0.0:1234go to http://localhost:1234
Use
creating data sets
- make a hand sign
- click the button to create training data
- it will wait 1 second, start recording data for 30 seconds, then save a file with the data