Defoe: A spark-based toolbox for analysing digital historical textual data

Rosa Filgueira; Michael Jackson; Anna Roubickova; Amrey Krause; Ruth Ahnert; Tessa Hauswedell; Julianne Nyhan; David Beavan; Timothy Hobson; Mariona Coll Ardanuy; Giovanni Colavizza; James Hetherington; Melissa Terras

doi:10.1109/eScience.2019.00033

Defoe: A spark-based toolbox for analysing digital historical textual data

Rosa Filgueira, Michael Jackson, Anna Roubickova, Amrey Krause, Ruth Ahnert, Tessa Hauswedell, Julianne Nyhan, David Beavan, Timothy Hobson, Mariona Coll Ardanuy, Giovanni Colavizza, James Hetherington, Melissa Terras

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.

Original language	English
Title of host publication	Proceedings - IEEE 15th International Conference on eScience, eScience 2019
Publisher	IEEE
Pages	235-242
Number of pages	8
ISBN (Electronic)	9781728124513
DOIs	https://doi.org/10.1109/eScience.2019.00033
Publication status	Published - Sept 2019
Event	15th IEEE International Conference on eScience 2019 - San Diego, United States Duration: 24 Sept 2019 → 27 Sept 2019

Conference

Conference	15th IEEE International Conference on eScience 2019
Abbreviated title	eScience 2019
Country/Territory	United States
City	San Diego
Period	24/09/19 → 27/09/19

Keywords

Apache Spark
Digital tools
Distributed queries
High-Performance Computing
Historical sources
Humanities research
Text mining
XML schemas

ASJC Scopus subject areas

Computer Science Applications
Software
Ecological Modelling
Modelling and Simulation

Access to Document

10.1109/eScience.2019.00033

Cite this

Filgueira, R., Jackson, M., Roubickova, A., Krause, A., Ahnert, R., Hauswedell, T., Nyhan, J., Beavan, D., Hobson, T., Coll Ardanuy, M., Colavizza, G., Hetherington, J., & Terras, M. (2019). Defoe: A spark-based toolbox for analysing digital historical textual data. In Proceedings - IEEE 15th International Conference on eScience, eScience 2019 (pp. 235-242). IEEE. https://doi.org/10.1109/eScience.2019.00033

@inproceedings{ee827173db584b089c5521eb42c8c2e6,

title = "Defoe: A spark-based toolbox for analysing digital historical textual data",

abstract = "This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.",

keywords = "Apache Spark, Digital tools, Distributed queries, High-Performance Computing, Historical sources, Humanities research, Text mining, XML schemas",

author = "Rosa Filgueira and Michael Jackson and Anna Roubickova and Amrey Krause and Ruth Ahnert and Tessa Hauswedell and Julianne Nyhan and David Beavan and Timothy Hobson and {Coll Ardanuy}, Mariona and Giovanni Colavizza and James Hetherington and Melissa Terras",

note = "Funding Information: ACKNOWLEDGEMENTS This work was funded by Scottish Enterprise as part of the Alan Turing Institute-Scottish Enterprise Data Engineering Programme, and by AHRC as part of the Living with Machines via the Strategic Priorities Fund. Publisher Copyright: {\textcopyright} 2019 IEEE.; 15th IEEE International Conference on eScience 2019, eScience 2019 ; Conference date: 24-09-2019 Through 27-09-2019",

year = "2019",

month = sep,

doi = "10.1109/eScience.2019.00033",

language = "English",

pages = "235--242",

booktitle = "Proceedings - IEEE 15th International Conference on eScience, eScience 2019",

publisher = "IEEE",

address = "United States",

}

Filgueira, R, Jackson, M, Roubickova, A, Krause, A, Ahnert, R, Hauswedell, T, Nyhan, J, Beavan, D, Hobson, T, Coll Ardanuy, M, Colavizza, G, Hetherington, J & Terras, M 2019, Defoe: A spark-based toolbox for analysing digital historical textual data. in Proceedings - IEEE 15th International Conference on eScience, eScience 2019. IEEE, pp. 235-242, 15th IEEE International Conference on eScience 2019, San Diego, California, United States, 24/09/19. https://doi.org/10.1109/eScience.2019.00033

TY - GEN

T1 - Defoe

T2 - 15th IEEE International Conference on eScience 2019

AU - Filgueira, Rosa

AU - Jackson, Michael

AU - Roubickova, Anna

AU - Krause, Amrey

AU - Ahnert, Ruth

AU - Hauswedell, Tessa

AU - Nyhan, Julianne

AU - Beavan, David

AU - Hobson, Timothy

AU - Coll Ardanuy, Mariona

AU - Colavizza, Giovanni

AU - Hetherington, James

AU - Terras, Melissa

N1 - Funding Information: ACKNOWLEDGEMENTS This work was funded by Scottish Enterprise as part of the Alan Turing Institute-Scottish Enterprise Data Engineering Programme, and by AHRC as part of the Living with Machines via the Strategic Priorities Fund. Publisher Copyright: © 2019 IEEE.

PY - 2019/9

Y1 - 2019/9

N2 - This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.

AB - This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.

KW - Apache Spark

KW - Digital tools

KW - Distributed queries

KW - High-Performance Computing

KW - Historical sources

KW - Humanities research

KW - Text mining

KW - XML schemas

UR - http://www.scopus.com/inward/record.url?scp=85083244614&partnerID=8YFLogxK

U2 - 10.1109/eScience.2019.00033

DO - 10.1109/eScience.2019.00033

M3 - Conference contribution

AN - SCOPUS:85083244614

SP - 235

EP - 242

BT - Proceedings - IEEE 15th International Conference on eScience, eScience 2019

PB - IEEE

Y2 - 24 September 2019 through 27 September 2019

ER -

Defoe: A spark-based toolbox for analysing digital historical textual data

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this