The language of AI and human poetry: a comparative lexicometric study

Abstract

This study conducts a lexicometric analysis to compare the lexical richness and diversity in poetry generated by AI models with that of human poets. Employing a robust dataset that includes 1,333 AI-generated poems and 517 humanauthored poems across seven distinct poetic eras, six key lexical metrics—Maas Index, MTLD, MATTR, HD-D, Hapax Legomenon Ratio, and Lexical Density—were applied for comparative analysis. The lexical characteristics of the poems were studied through a series of statistical tests and machine learning techniques, including Mann-Whitney U tests, Cliff's Delta, and Random Forest classification. The findings reveal a marked lexical superiority in human poetry, evidenced by significant differences and large effect sizes in all metrics except Lexical Density. HD-D emerged as the most discriminating factor, adeptly differentiating human poetry from its AI-generated counterpart. Further analysis identified the GPT-4 model as exhibiting the closest alignment to human poetry in terms of lexical attributes. The study discusses these outcomes in the context of AI's evolving linguistic competencies, shedding light on the inherent challenges and future prospects of AI in creative writing. Thus, this research provides an empirical framework for assessing AI’s language generation abilities and sets the stage for further interdisciplinary exploration into the frontiers of artificial creativity.