Abstract
We propose a novel automatic evaluation metric for open-ended text generation, which is a substantial improvement of the recently developed method, Fourier analysis of cross-entropy (FACE), hence, FACE-2. FACE-2 is a psycholinguistically inspired metric that extracts the dynamic patterns (spectrum) of text surprisal. Examined with open-ended text generation tasks, FACE-2 significantly outperforms a broad set of baseline metrics in revealing the model scaling effect, which scales up to models of 70B parameters, while many other existing metrics fail to capture this effect. We have also confirmed the advantage of FACE-2 in producing stronger agreement with human preferences from a large human-annotated dataset. We advocate for including metrics that mine the dynamics of likelihood in evaluating open-ended text generation, which covers broader aspects of human language than only using static likelihood-based or semantic-based metrics. Code repository: https://github.com/CLCS-SUSTech/FACEScore.
| Original language | English |
|---|---|
| Title of host publication | Findings of the Association for Computational Linguistics: EMNLP 2025 |
| Editors | Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng |
| Publisher | Association for Computational Linguistics |
| Pages | 2444-2463 |
| Number of pages | 20 |
| ISBN (Electronic) | 9798891763357 |
| DOIs | |
| Publication status | Published - Nov 2025 |
| Event | 30th Conference on Empirical Methods in Natural Language Processing 2025 - Suzhou, China Duration: 4 Nov 2025 → 9 Nov 2025 |
Conference
| Conference | 30th Conference on Empirical Methods in Natural Language Processing 2025 |
|---|---|
| Abbreviated title | EMNLP 2025 |
| Country/Territory | China |
| City | Suzhou |
| Period | 4/11/25 → 9/11/25 |
ASJC Scopus subject areas
- Computational Theory and Mathematics
- Computer Science Applications
- Information Systems
- Linguistics and Language
Fingerprint
Dive into the research topics of 'Evaluating Text Generation Quality Using Spectral Distances of Surprisal'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver