Record something, play the on-screen keyboard, or type lyrics. HearMe returns an arrangement built around it. The stems come back separate so you can solo and mute them, with a chord track and lyrics lined up to the same timeline, and a play-along mode on top.
What the models do
The generative model runs as a service at the Center for Music Technology. The client sends up a performance: recorded audio, MIDI captured from the keyboard, or typed lyrics. Nobody types a prompt. The conditioning signal is whatever the musician actually played, which is the premise the whole app is built on.
Chord estimation and lyric alignment come back with the audio. Both are music information retrieval problems in their own right, and the client has to trust their timing closely enough that a musician can scrub, loop and play along against them.
I work on the client. Most of the engineering sits in the gaps around inference. A generation takes long enough that you cannot leave someone watching a spinner, so the request goes to a background watcher that survives the app being suspended and reports back when the result lands. The stems arrive as separate files and have to be scheduled sample-accurately against one transport, or they drift away from the chord and lyric tracks they are supposed to agree with. Results are versioned per project, so two attempts at the same idea can sit side by side.
The client
- Web-Audio-style playback engine in React Native with sample-accurate scheduling for the piano and drum interfaces, plus offline MIDI rendering of results.
- Full input path into the generation service: recording, upload, keyboard capture, lyric entry. A background watcher survives app suspension.
- Result view: one transport across stems, chords, lyrics and play-along, with per-project version history.
React Native · Expo · react-native-audio-api · Skia · Appwrite