Do Semantic Distance Tests Actually Predict Creativity in LLMs? Evidence for Jagged Creativity

원문 초록

Measuring the creativity of large language models (LLMs) is essential for designing methods that can improve creativity and for enhancing our scientific understanding of this ability. To accomplish this, it has become common in recent years to administer semantic distance tests of human creativity to LLMs. Although these tests provide a convenient and fully automated way to score aspects of "creativity," their validity as measures of machine creativity has not been established. To address this problem, we conduct the first large-scale, systematic study assessing the effectiveness of semantic distance tests for predicting the creative achievement of LLMs across four target constructs: (a) creative writing, (b) output diversity on open-ended prompts, (c) combinatorial creativity, and (d) scientific ideation. We score each test on two criteria (validity and specificity), and derive a theoretical limit for the maximum attainable specificity and validity a test can achieve. We find that: (1) No test is a significant predictor of output diversity, suggesting that point creativity, the ability to produce a single response connecting semantically distant concepts, and distributional creativity, the ability to produce distinct responses across repeated samples, may dissociate in current LLMs--a pattern we call jagged creativity. (2) The Remote Associates Test is effectively a proxy for general capabilities rather than a specific predictor of creativity in LLMs. (3) Lastly, the Divergent Association Task is the best predictor of creative writing, but no single test predicts all constructs well. In summary, our results prescribe caution when applying psychometric tests and tasks of creativity validated among human populations to LLMs.