Skip to main content

Making State Explicit for Imperative Big Data Processing

Castro Fernandez, Raul and Migliavacca, Matteo and Kalyvianaki, Evangelia and Pietzuch, Peter (2014) Making State Explicit for Imperative Big Data Processing. In: 2nd Symposium on Networked Systems Design & Implementation 2005. USENIX Association. ISBN 978-1-931971-10-2.

Abstract

Data scientists often implement machine learning algorithms in imperative languages such as Java, Matlab and R. Yet such implementations fail to achieve the performance and scalability of specialised data-parallel processing frameworks. Our goal is to execute imperative Java programs in a data-parallel fashion with high throughput and low latency. This raises two challenges: how to support the arbitrary mutable state of Java programs without compromising scalability, and how to recover that state after failure with low overhead.

Our idea is to infer the dataflow and the types of state accesses from a Java program and use this information to generate a stateful dataflow graph (SDG). By explicitly separating data from mutable state, SDGs have specific features to enable this translation: to ensure scalability, distributed state can be partitioned across nodes if computation can occur entirely in parallel; if this is not possible, partial state gives nodes local instances for independent computation, which are reconciled according to application semantics. For fault tolerance, large inmemory state is checkpointed asynchronously without global coordination. We show that the performance of SDGs for several imperative online applications matches that of existing data-parallel processing frameworks.

Item Type: Book section
Subjects: Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming,
Divisions: Faculties > Sciences > School of Computing > Data Science
Depositing User: Matteo Migliavacca
Date Deposited: 09 Jul 2015 11:26 UTC
Last Modified: 24 Jan 2020 04:07 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/49288 (The current URI for this page, for reference purposes)
Migliavacca, Matteo: https://orcid.org/0000-0002-5684-4865
  • Depositors only (login required):

Downloads

Downloads per month over past year