Waifu Synthesis (2019) is a bit of a playful project investigating real-time generation of singing anime characters, a neural mashup if you will.
All of the animation is made in real-time using a StyleGan neural network
trained on the Danbooru2018 dataset, a large scale anime image database
with 3.33m+ images annotated with 99.7m+ tags.
Lyrics were produced with GPT-2, a large scale language model trained on
40GB of internet text. The project uses the recently released 345 million
parameter version- the full model has 1.5 billion parameters, and has currently not been released due to concerns about malicious use (think fake news).
Music was made in part using models from Magenta, a research project
exploring the role of machine learning in the process of creating art and music.
Setup is using vvvv, Python and Ableton Live.
StyleGan, Danbooru2018, GPT-2 and Magenta were developed by Nvidia,
gwern.net/Danbooru2018, OpenAI and Google respectively.
