Effects of Visual Cues on Musical Melody Imagery for EEG-based Brain-Computer Interfaces

Abstract

Decoding imagined musical pitch from electroencephalography (EEG) remains a challenge in the development of music brain-computer interfaces (BCIs), primarily due to ambiguous and inconsistent EEG responses to music imagery. Based on findings in motor imagery BCIs showing that different sensory modalities such as visual or kinesthetic cues can foster motor imagery, this study investigates whether visual cues facilitate music imagery and the reliability of EEG responses. We compared EEG responses across several types of visual cues during both the perception and imagination of musical pitch sequences. Ten musically trained participants listened to five-tone melodies generated from seven piano tones in the 4th octave (C4-B4; Do, Re, Mi, Fa So, La, and Ti), followed by the imagination of humming the same melodies. Each melody was structured using harmonic rules and Hanon-inspired ascending or descending pitch patterns. Each trial consisted of two tasks: (1) a perception task, where participants heard a melody while one of the four visual cues was presented—a keyboard image, letter name, musical notation, or a static note symbol. All cues except the static note symbol were animated to be semantically congruent with the melody. (2) A subsequent imagery task, where participants imagined the same melody while viewing only the static symbol, regardless of the preceding visual cue. This design allowed us to investigate which types of visual cues best support musical imagery while avoiding confounds during the imagery task itself. During the perception task, event-related potentials (ERPs) were modulated by pitch height, with keyboard inducing more distinct ERP responses than others. Notably, even in the imagery task—where participants performed imagery with only the static symbol—ERP waveforms varied with pitch height. Interestingly, when the keyboard cue was presented during perception, the subsequent imagery task evoked ERP waveforms with reversed polarity compared to those during the perception task. This result aligns with prior findings that perception and imagery can evoke opposite neural dynamics, and further suggests that keyboard cue may amplify such contrasts. Post-experiment survey feedback also supported this implication, as participants reported that melodies paired with the keyboard cue were easier to imagine, possibly contributing to stronger internal pitch representation. However, the keyboard cue can induce lateral eye movements, emphasizing the need for artifact removal and careful visual cue design. Our results may provide insights for the utilization of visual cues in music BCIs to improve the decoding of imagined pitch.