Skip to main content

Making State Explicit for Imperative Big Data Processing

Castro Fernandez, Raul and Migliavacca, Matteo and Kalyvianaki, Evangelia and Pietzuch, Peter (2014) Making State Explicit for Imperative Big Data Processing. In: 2014 USENIX Annual Technical Conference (USENIX ATC 14). USENIX Association. ISBN 978-1-931971-10-2. (KAR id:49288)


Data scientists often implement machine learning algorithms in imperative languages such as Java, Matlab and R. Yet such implementations fail to achieve the performance and scalability of specialised data-parallel processing frameworks. Our goal is to execute imperative Java programs in a data-parallel fashion with high throughput and low latency. This raises two challenges: how to support the arbitrary mutable state of Java programs without compromising scalability, and how to recover that state after failure with low overhead.

Our idea is to infer the dataflow and the types of state accesses from a Java program and use this information to generate a stateful dataflow graph (SDG). By explicitly separating data from mutable state, SDGs have specific features to enable this translation: to ensure scalability, distributed state can be partitioned across nodes if computation can occur entirely in parallel; if this is not possible, partial state gives nodes local instances for independent computation, which are reconciled according to application semantics. For fault tolerance, large inmemory state is checkpointed asynchronously without global coordination. We show that the performance of SDGs for several imperative online applications matches that of existing data-parallel processing frameworks.

Item Type: Book section
Subjects: Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming,
Divisions: Faculties > Sciences > School of Computing > Data Science
Depositing User: Matteo Migliavacca
Date Deposited: 09 Jul 2015 11:26 UTC
Last Modified: 15 Oct 2020 10:15 UTC
Resource URI: (The current URI for this page, for reference purposes)
Migliavacca, Matteo:
  • Depositors only (login required):


Downloads per month over past year