Zhihao Ouyang, Yihang Yin, Kun Yan, Jian Wu, Xiaolin Hu

XiaoJoy: Using Locally Connected CNN to Cooperate with Human Composers (2018)

Recently, lots of work has been done to build AI composers. However, without human involvement, these generated music couldn’t get over with rigidity. Thus, we want to deepen the interactions between human and AI(XiaoJoy).  XiaoJoy is an AI system which mainly consists of two models. One is a Convolutional Neural Network(CNN) model, and another is a revised version of Google Magenta Improved RNN model (rGMIR).  As for CNN model, it can generate a single-track melody when given a simple melody as input. We use Locally Connected Convolution blocks to replace general CNN blocks on ResNet structure. In rGMIR model, it receives both melody and chords as the input, that means composer can stipulate the chord progression in music. We trained CNN models on Wikifonia dataset which contains 6,675 xml music pieces. To let both 2 models generates similar music, we use the pre-trained model from Google Magenta project and fine-tune rGMIR model on the Wikifonia dataset. We split the music composition task into 2 parts: composing main melody and accompaniment. XiaoJoy takes charge of the former where the composer performs the latter. In our video, XiaoJoy’s CNN model generates 2 samples using the Little Star as input, and XiaoJoy’s rGMIR model generates 4 samples with both the Canon’s melody and Canon’s Chords as input. The human composer then finishes 6 accompaniment parts by referencing the main melody generated by XiaoJoy. The violinist plays the main melody generated by XiaoJoy, and the accompaniment is rendered on Logic Pro X. As to the CNN model, because it doesn’t receive any supervisory information, so composers have to totally ”follow” its generation result. However when it comes to rGMIR model, the model will generate melody based on Canon’s chord progression, composers get much easier to “cooperate” with AI, making it easier to harmonize the melody. Overall, our demo illustrates a framework for musicians to cooperate with AI. Although, for now, AI won’t surpass the quality of professional musicians, it can provide hundreds of alternative pieces instantly. Moreover, with more supervised music information like chords, human and AI will be able to understand each other’s intention. This usage of AI shifts human’s work from composing to choosing and complementing, which also brings writing personal music to non-specialists. We believe human’s judgement, together with AI’s efficiency, will certainly bring new possibilities to the realm of music.

Website Link: https://github.com/Somedaywilldo/local_conv_music_generation