Dispel4py: A python framework for data-intensive scientific computing

Rosa Filguiera; Iraklis Klampanos; Amrey Krause; Mario David; Alexander Moreno; Malcolm Atkinson

doi:10.1109/DISCS.2014.12

Dispel4py: A python framework for data-intensive scientific computing

Rosa Filguiera, Iraklis Klampanos, Amrey Krause, Mario David, Alexander Moreno, Malcolm Atkinson

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

14 Citations (Scopus)

Abstract

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. The main aim of dispel4py is to enable scientists to focus on their computation instead of being distracted by details of the computing infrastructure they use. Therefore, special care has been taken to provide dispel4py with the ability to map abstract workflows to different enactment platforms dynamically, at run time. In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential. The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.

Original language	English
Title of host publication	2014 International Workshop on Data Intensive Scalable Computing Systems
Publisher	IEEE
Pages	9-16
Number of pages	8
ISBN (Electronic)	9781479970384
DOIs	https://doi.org/10.1109/DISCS.2014.12
Publication status	Published - 6 Apr 2014
Event	2014 International Workshop on Data-Intensive Scalable Computing Systems - New Orleans, United States Duration: 16 Nov 2014 → …

Conference

Conference	2014 International Workshop on Data-Intensive Scalable Computing Systems
Abbreviated title	DISCS 2014
Country/Territory	United States
City	New Orleans
Period	16/11/14 → …

Keywords

data streaming
Data-intensive computing
e-Infrastructures
programming frameworks
Python
scientific workflows

ASJC Scopus subject areas

Software
Information Systems
Computer Networks and Communications

Access to Document

10.1109/DISCS.2014.12

Cite this

@inproceedings{19cf05f5d1e047dea2dd2ebb2138f9c3,

title = "Dispel4py: A python framework for data-intensive scientific computing",

abstract = "This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. The main aim of dispel4py is to enable scientists to focus on their computation instead of being distracted by details of the computing infrastructure they use. Therefore, special care has been taken to provide dispel4py with the ability to map abstract workflows to different enactment platforms dynamically, at run time. In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential. The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.",

keywords = "data streaming, Data-intensive computing, e-Infrastructures, programming frameworks, Python, scientific workflows",

author = "Rosa Filguiera and Iraklis Klampanos and Amrey Krause and Mario David and Alexander Moreno and Malcolm Atkinson",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.; 2014 International Workshop on Data-Intensive Scalable Computing Systems, DISCS 2014 ; Conference date: 16-11-2014",

year = "2014",

month = apr,

day = "6",

doi = "10.1109/DISCS.2014.12",

language = "English",

pages = "9--16",

booktitle = "2014 International Workshop on Data Intensive Scalable Computing Systems",

publisher = "IEEE",

address = "United States",

}

Filguiera, R, Klampanos, I, Krause, A, David, M, Moreno, A & Atkinson, M 2014, Dispel4py: A python framework for data-intensive scientific computing. in 2014 International Workshop on Data Intensive Scalable Computing Systems. IEEE, pp. 9-16, 2014 International Workshop on Data-Intensive Scalable Computing Systems, New Orleans, United States, 16/11/14. https://doi.org/10.1109/DISCS.2014.12

TY - GEN

T1 - Dispel4py

T2 - 2014 International Workshop on Data-Intensive Scalable Computing Systems

AU - Filguiera, Rosa

AU - Klampanos, Iraklis

AU - Krause, Amrey

AU - David, Mario

AU - Moreno, Alexander

AU - Atkinson, Malcolm

PY - 2014/4/6

Y1 - 2014/4/6

N2 - This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. The main aim of dispel4py is to enable scientists to focus on their computation instead of being distracted by details of the computing infrastructure they use. Therefore, special care has been taken to provide dispel4py with the ability to map abstract workflows to different enactment platforms dynamically, at run time. In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential. The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.

AB - This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. The main aim of dispel4py is to enable scientists to focus on their computation instead of being distracted by details of the computing infrastructure they use. Therefore, special care has been taken to provide dispel4py with the ability to map abstract workflows to different enactment platforms dynamically, at run time. In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential. The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.

KW - data streaming

KW - Data-intensive computing

KW - e-Infrastructures

KW - programming frameworks

KW - Python

KW - scientific workflows

UR - http://www.scopus.com/inward/record.url?scp=84946531115&partnerID=8YFLogxK

U2 - 10.1109/DISCS.2014.12

DO - 10.1109/DISCS.2014.12

M3 - Conference contribution

AN - SCOPUS:84946531115

SP - 9

EP - 16

BT - 2014 International Workshop on Data Intensive Scalable Computing Systems

PB - IEEE

Y2 - 16 November 2014

ER -

Dispel4py: A python framework for data-intensive scientific computing

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this