On the power of special-purpose GPT models to create and evaluate new poetry in old styles

This study investigates the possibility of using GPT-3 models to generate high-quality poems in a specific author's style, through fine-tuning on datasets of poems accompanied by their metadata and automatically generated summaries. Our experiments show that a dataset of only 300 poems is sufficient to generate new poems in the style of a specific author. The evaluation was done through GPT-3 models fine-tuned for binary classification of GPT-3 outputs against the works of the original author. To establish the accuracy of GPT-3-based binary classifiers, we first tested them on a variety of texts and a range of classes, and found that their predictive accuracy is 99% on average. Using this method for poetry evaluation showed that the GPT-3 generated poems were indistinguishable from the original works of Walt Whitman and Rudyard Kipling in an average of 30% and 21% of the cases, respectively. This suggests that GPT-3 can be a useful tool in assisting authors, while further research is needed to turn it into an independent creator. Additionally, the workflow used in this study can be applied to other types of text and provides a way of using GPT-3 models for generating new content from user-provided summaries, when prompt engineering alone is insufficient.