For this research project we explored creative generative models for text (poems, lyrics, metaphors) with three main components: (1) Pre-train on large-scale creative text (Gutenberg novels), (2) implement AWD-LSTM [1] and Transformer XL-based [2] Generative Adversarial Networks with a Discriminator feedback that evaluates if a piece of text is creative or non-creative and (3) substitute maximum likelihood optimization (MLE) and optimize for creative rather than the most likely tokens to elicit unique word combinations.

​

Just as the results of a human creative process, not all outputs of our generative models are great. But every once in a while we see some AI Fragments, that are unusual, strange, funny, deep, and definitely creative. Some curated creative outputs are displayed on this website. 

​

Content curated and website made by Suzana Ilić.

​

For technical details please refer to our paper:

​

Asir Saeed, Suzana Ilić, and Eva Zangerle: Creative GANs for generating poems, lyrics, and metaphors. arXiv preprint, 2019.

​

Abstract

Generative models for text have substantially contributed to tasks like machine translation and language modeling, using maximum likelihood optimization (MLE). However, for creative text generation, where multiple outputs are possible and originality and uniqueness are encouraged, MLE falls short. Methods optimized for MLE lead to outputs that can be generic, repetitive and incoherent. In this work, we use a Generative Adversarial Network framework to alleviate this problem. We evaluate our framework on poetry, lyrics and metaphor datasets, each with widely different characteristics, and report better performance of our objective function over other generative models.

​

If you'd like to share feedback, please visit our GitHub repository or contact the authors directly.

​

MLT LABS is part of Machine Learning Tokyo, a Japan-based nonprofit organization working on open source, education and research projects in Machine Learning, Deep Learning and Robotics. 

​

​

References

 

[1] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182, 2017

​

[2] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le,and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-lengthcontext.arXiv preprint arXiv:1901.02860, 2019.