Skip to main navigation Skip to search Skip to main content

Evaluating Text Generation Quality Using Spectral Distances of Surprisal

  • Zhichen Liu
  • , Yongyuan Li
  • , Yang Xu*
  • , Yu Wang
  • , Yingfang Yuan
  • , Zuhao Yang
  • *Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Downloads (Pure)

Abstract

We propose a novel automatic evaluation metric for open-ended text generation, which is a substantial improvement of the recently developed method, Fourier analysis of cross-entropy (FACE), hence, FACE-2. FACE-2 is a psycholinguistically inspired metric that extracts the dynamic patterns (spectrum) of text surprisal. Examined with open-ended text generation tasks, FACE-2 significantly outperforms a broad set of baseline metrics in revealing the model scaling effect, which scales up to models of 70B parameters, while many other existing metrics fail to capture this effect. We have also confirmed the advantage of FACE-2 in producing stronger agreement with human preferences from a large human-annotated dataset. We advocate for including metrics that mine the dynamics of likelihood in evaluating open-ended text generation, which covers broader aspects of human language than only using static likelihood-based or semantic-based metrics. Code repository: https://github.com/CLCS-SUSTech/FACEScore.

Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: EMNLP 2025
EditorsChristos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng
PublisherAssociation for Computational Linguistics
Pages2444-2463
Number of pages20
ISBN (Electronic)9798891763357
DOIs
Publication statusPublished - Nov 2025
Event30th Conference on Empirical Methods in Natural Language Processing 2025 - Suzhou, China
Duration: 4 Nov 2025 → 9 Nov 2025

Conference

Conference30th Conference on Empirical Methods in Natural Language Processing 2025
Abbreviated titleEMNLP 2025
Country/TerritoryChina
CitySuzhou
Period4/11/25 → 9/11/25

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Information Systems
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'Evaluating Text Generation Quality Using Spectral Distances of Surprisal'. Together they form a unique fingerprint.

Cite this