The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task

Roy Schwartz; Maarten Sap; Ioannis Konstas; Li Zilles; Yejin Choi; Noah A. Smith

doi:10.18653/v1/k17-1004

The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task

Roy Schwartz, Maarten Sap, Ioannis Konstas, Li Zilles, Yejin Choi, Noah A. Smith

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

70 Citations (Scopus)

Abstract

A writer’s style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.¹

Original language	English
Title of host publication	Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)
Publisher	Association for Computational Linguistics
Pages	15-25
Number of pages	11
ISBN (Electronic)	9781945626548
DOIs	https://doi.org/10.18653/v1/k17-1004
Publication status	Published - Aug 2017
Event	21st Conference on Computational Natural Language Learning 2017 - Vancouver, Canada Duration: 3 Aug 2017 → 4 Aug 2017

Conference

Conference	21st Conference on Computational Natural Language Learning 2017
Abbreviated title	CoNLL 2017
Country/Territory	Canada
City	Vancouver
Period	3/08/17 → 4/08/17

ASJC Scopus subject areas

Linguistics and Language
Artificial Intelligence
Human-Computer Interaction

Access to Document

10.18653/v1/k17-1004Licence: CC BY

Cite this

@inproceedings{0531f8934a484fb4840b0cdd34a725aa,

title = "The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task",

abstract = "A writer{\textquoteright}s style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.1",

author = "Roy Schwartz and Maarten Sap and Ioannis Konstas and Li Zilles and Yejin Choi and Smith, \{Noah A.\}",

note = "Publisher Copyright: {\textcopyright} 2017 Association for Computational Linguistics.; 21st Conference on Computational Natural Language Learning 2017, CoNLL 2017 ; Conference date: 03-08-2017 Through 04-08-2017",

year = "2017",

month = aug,

doi = "10.18653/v1/k17-1004",

language = "English",

pages = "15--25",

booktitle = "Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)",

publisher = "Association for Computational Linguistics",

}

Schwartz, R, Sap, M, Konstas, I, Zilles, L, Choi, Y & Smith, NA 2017, The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. in Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Association for Computational Linguistics, pp. 15-25, 21st Conference on Computational Natural Language Learning 2017, Vancouver, British Columbia, Canada, 3/08/17. https://doi.org/10.18653/v1/k17-1004

The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. / Schwartz, Roy; Sap, Maarten; Konstas, Ioannis et al.
Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Association for Computational Linguistics, 2017. p. 15-25.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - The effect of different writing tasks on linguistic style

T2 - 21st Conference on Computational Natural Language Learning 2017

AU - Schwartz, Roy

AU - Sap, Maarten

AU - Konstas, Ioannis

AU - Zilles, Li

AU - Choi, Yejin

AU - Smith, Noah A.

PY - 2017/8

Y1 - 2017/8

N2 - A writer’s style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.1

AB - A writer’s style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.1

UR - https://www.scopus.com/pages/publications/85063089960

U2 - 10.18653/v1/k17-1004

DO - 10.18653/v1/k17-1004

M3 - Conference contribution

AN - SCOPUS:85063089960

SP - 15

EP - 25

BT - Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)

PB - Association for Computational Linguistics

Y2 - 3 August 2017 through 4 August 2017

ER -

The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task

Abstract

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this