Dispel4py: A python framework for data-intensive eScience

Amrey Krause, Rosa Filgueira, Malcolm Atkinson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present dispel4py, a novel data intensive and high performance computing middleware provided as a standard Python library for describing stream-based workows. It allows its users to develop their scientific applications locally and then run them on a wide range of HPC-infrastructures without any changes to the code. Moreover, it provides automated and efficient parallel mappings toMPI, multiprocessing, Storm and Spark frameworks, commonly used in big data applications. It builds on the wide availability of Python in many environments and only requires familiarity with basic Python syntax. We will show the dispel4py advantages by walking through an example. We will conclude demonstrating how dispel4py can be employed as an easy-to-use tool for designing scientific applications using real-world scenarios.

Original languageEnglish
Title of host publicationPyHPC '15: Proceedings of the 5th Workshop on Python for High-Performance and Scientific Computing
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450340106
DOIs
Publication statusPublished - 15 Nov 2015
Event5th Workshop on Python for High-Performance and Scientific Computing 2015 - Austin, United States
Duration: 15 Nov 2015 → …

Conference

Conference5th Workshop on Python for High-Performance and Scientific Computing 2015
Abbreviated titlePyHPC 2015
CountryUnited States
CityAustin
Period15/11/15 → …

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'Dispel4py: A python framework for data-intensive eScience'. Together they form a unique fingerprint.

Cite this