On the Creative Application of Large Language Models

Large language models (LLMs) have become a common sight in day-to-day activities and have found use for a wide range of tasks, many of them of a creative nature. While not perfect, these models have enabled automation of tasks that were previously difficult to automate, such as generating fluent narratives or using them as creative evaluators. But to what extent can pure large language models perform creative tasks? This dissertation explores and evaluates the use of LLMs without augmentations for creative applications, specifically in the domain of narrative generation. The first contribution of this dissertation is an analysis of LLMs and their creative po- tential, including evaluation using three established CC evaluation frameworks: "Mere Generation", The Creative Tripod, and FACE. I argue that it is inappropriate to evaluate LLMs using these frameworks, primarily because their "process" is simple, yet they can perform complex tasks. Moreover, these frameworks often require explicitly modelled features to argue for the creativity of the system. However, for LLMs, their abilities are implicit in their learned parameters. Secondly, I shift focus to empirical evaluation to investigate a simple yet common claim in popular media and academic literature: is temperature the universal creativity parameter of pure large language models? I have designed a novel experimental method based on theories from cognitive science, and reframed the temperature zero story as the exemplar story for a given prompt. This exemplar enables comparison in a human evaluation experiment to understand the impact of the temperature hyperparameter. The results indicate that this claim is far more nuanced and weaker than is often presented. It appears higher temperature enable more exploration by increasing the chance of generating more variety, but it does not allow access to a significantly larger slice of the probability distribution. This observation raises questions regarding the output diversity of LLMs, and particularly, those trained for instruction following. The third contribution of this dissertation is two- fold. First, I investigate the gap in output diversity between instruction-tuned models and their corresponding base models. While this phenomenon has been observed before, I confirm its existence in the domain of narrative generation and demonstrate further findings on how intermediate fine-tuning steps impact the output diversity. Secondly, I present a novel decoding strategy, conformative decoding, motivated by the diversity gap that aims to reintroduce some of the lost output diversity. Conformative decoding works by mixing the next-token distribution of the base model into the next-token distribution of the instruction-tuned model. The computational results show a slight increase in output diversity. In contrast, the human evaluation experiment reveals no preference for either the baseline or conformative decoding. Assessing the diversity of a set of texts is a complex task, and the difference in diversity may be too subtle to detect or not meaningful to humans. LLMs are powerful tools, and we have only begun understanding their creative potential. The findings in this dissertation show the limits of pure LLMs performing creative tasks and our ability to evaluate them using current methods. However, based on the findings, I believe there is much potential for future work on using multi-agent setups that view creativity as a social phenomenon.