Dispel4py: An agile framework for data-intensive eScience

Rosa Filgueira, Amrey Krause, Malcolm Atkinson, Iraklis Klampanos, Alessandro Spinuso, Susana Sanchez-Exposito

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Citations (Scopus)

Abstract

We present dispel4py a versatile data-intensive kit presented as a standard Python library. It empowers scientists to experiment and test ideas using their familiar rapid-prototyping environment. It delivers mappings to diverse computing infrastructures, including cloud technologies, HPC architectures and specialised data-intensive machines, to move seamlessly into production with large-scale data loads. The mappings are fully automated, so that the encoded data analyses and data handling are completely unchanged. The underpinning model is lightweight composition of fine-grained operations on data, coupled together by data streams that use the lowest cost technology available. These fine-grained workflows are locally interpreted during development and mapped to multiple nodes and systems such as MPI and Storm for production. We explain why such an approach is becoming more essential in order that data-driven research can innovate rapidly and exploit the growing wealth of data while adapting to current technical trends. We show how provenance management is provided to improve understanding and reproducibility, and how a registry supports consistency and sharing. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved. Finally we present the next steps to achieve scalability and performance.

Original languageEnglish
Title of host publication2015 IEEE 11th International Conference on e-Science
PublisherIEEE
Pages454-464
Number of pages11
ISBN (Electronic)9781467393256
DOIs
Publication statusPublished - 26 Oct 2015
Event11th IEEE International Conference on eScience 2015 - Munich, Germany
Duration: 31 Aug 20154 Sept 2015

Conference

Conference11th IEEE International Conference on eScience 2015
Abbreviated titleeScience 2015
Country/TerritoryGermany
CityMunich
Period31/08/154/09/15

Keywords

  • Data intensive application
  • Distributed systems
  • ESciences workflows
  • Python frameworks
  • Run time analysis

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Management Science and Operations Research

Fingerprint

Dive into the research topics of 'Dispel4py: An agile framework for data-intensive eScience'. Together they form a unique fingerprint.

Cite this