Skip to main content

Making State Explicit for Imperative Big Data Processing

Castro Fernandez, Raul, Migliavacca, Matteo, Kalyvianaki, Evangelia, Pietzuch, Peter (2014) Making State Explicit for Imperative Big Data Processing. In: Proceedings of USENIX ATC '14: 2014 USENIX Annual Technical Conference. . pp. 49-60. USENIX Association ISBN 978-1-931971-10-2. (KAR id:49288)

PDF Publisher pdf
Language: English
Download (484kB) Preview
[thumbnail of atc14-paper-castro_fernandez.pdf]
This file may not be suitable for users of assistive technology.
Request an accessible format
Official URL


Data scientists often implement machine learning algorithms in imperative languages such as Java, Matlab and R. Yet such implementations fail to achieve the performance and scalability of specialised data-parallel processing frameworks. Our goal is to execute imperative Java programs in a data-parallel fashion with high throughput and low latency. This raises two challenges: how to support the arbitrary mutable state of Java programs without compromising scalability, and how to recover that state after failure with low overhead.

Our idea is to infer the dataflow and the types of state accesses from a Java program and use this information to generate a stateful dataflow graph (SDG). By explicitly separating data from mutable state, SDGs have specific features to enable this translation: to ensure scalability, distributed state can be partitioned across nodes if computation can occur entirely in parallel; if this is not possible, partial state gives nodes local instances for independent computation, which are reconciled according to application semantics. For fault tolerance, large inmemory state is checkpointed asynchronously without global coordination. We show that the performance of SDGs for several imperative online applications matches that of existing data-parallel processing frameworks.

Item Type: Conference or workshop item (Paper)
Subjects: Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming,
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Depositing User: Matteo Migliavacca
Date Deposited: 09 Jul 2015 11:26 UTC
Last Modified: 16 Feb 2021 13:26 UTC
Resource URI: (The current URI for this page, for reference purposes)
Migliavacca, Matteo:
  • Depositors only (login required):


Downloads per month over past year