Skip to main content
Kent Academic Repository

Training GPT-2 to represent two Romantic-era authors: challenges, evaluations and pitfalls

Sawicki, Piotr, Grzes, Marek, Jordanous, Anna, Brown, Dan, Peeperkorn, Max (2022) Training GPT-2 to represent two Romantic-era authors: challenges, evaluations and pitfalls. In: 13th International Conference on Computational Creativity. . pp. 34-43. Association for Computational Creativity (ACC) ISBN 978-989-54-1604-2. (KAR id:94992)


Poetry generation within style constraints has many creative challenges, despite the recent advances in Transformer models for text generation. We study 1) how overfitting of various versions of GPT-2 models affects the quality of the generated text, and 2) which model is better at generating text in a specific style. For that purpose, we propose a novel setup for text evaluation with neural networks. Our GPT-2 models are trained on datasets of collected works of the two Romantic-era poets: Byron and Shelley. With some models, overfitting manifests by producing malformed samples, with others, the samples are always well-formed, but contain increasingly higher levels of n-grams duplicated from the original corpus. This behaviour can lead to incorrect evaluations of generated text because the plagiarised output can deceive neural network classifiers and even human judges.

To determine which model is better at preserving style before it becomes overfitted, we conduct two series of experiments with BERT-based classifiers.

Overall, our results provide a novel way of selecting the right models for fine-tuning on a specific dataset, while highlighting the pitfalls that come with overfitting, like reordering and replicating text, towards more credible creative text generation.

Item Type: Conference or workshop item (Paper)
Subjects: Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming, > QA76.87 Neural computers, neural networks
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Funders: University of Kent (
Depositing User: Marek Grzes
Date Deposited: 11 May 2022 17:53 UTC
Last Modified: 06 Dec 2022 10:28 UTC
Resource URI: (The current URI for this page, for reference purposes)

University of Kent Author Information

  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.