Dispel4py: A Python framework for data-intensive scientific computing

Rosa Filguiera, Amrey Krause, Malcolm Atkinson, Iraklis Klampanos, Alexander Moreno

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)

Abstract

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map dispel4py abstract workflows optimally onto target platforms chosen dynamically. We present four dispel4py mappings: Apache Storm, message-passing interface (MPI), multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The dispel4py.org is an open-source project to which we invite participation. The effective mapping of dispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and high-performance computing (HPC) architectures and consistent scalability.

Original languageEnglish
Pages (from-to)316-334
Number of pages19
JournalInternational Journal of High Performance Computing Applications
Volume31
Issue number4
Early online date1 Jun 2017
DOIs
Publication statusPublished - 1 Jul 2017

Keywords

  • data streaming
  • Data-intensive computing
  • e-infrastructures
  • programming frameworks
  • scientific workflows

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Dispel4py: A Python framework for data-intensive scientific computing'. Together they form a unique fingerprint.

Cite this