Lingdong Huang, Zheng Jiang, Giada Sun, Tong Bai, Eunsu Kang, Barnabas Poczos


Legend of Wrong Mountain (2018). As one of the oldest forms of Chinese Opera since the 16th century, Kunqu features literary virtuosity in its scripts, sophisticated vocal techniques in its singing, emotional and elegant yet rigorous bodily motions and facial expressions in its performance. Over its history, Kunqu has developed established modes and patterns, which makes it especially suitable for neural networks to learn. In order to generate Kunqu literature and performance computationally, we applied machine learning techniques such as LSTM, pix2pix, pix2pixHD, RNN, Markov Chain, OpenPose and Detectron, as well as tools such as Canny Edge, PIL and SketchUp. Our result, Legend of Wrong Mountain, is novel twofold: it is the first and only generated Opera at the time of this writing, and it is a machine’s attempt at Gesamtkunstwerk, “the Total Artwork”. It explores the marriage between contemporary technologies and traditional art form. By studying historical scripts, musical notations and traditional methods for creating Kunqu, we tweaked existing algorithms and devised new ones to conform to the traditional rules and norms as closely as possible. We present this project as a video accompanied by audio.