Skip to main content

Spark on Entropy: A Reliable & Efficient Scheduler for Low-latency Parallel Jobs in Heterogeneous Cloud

Chen, Huankai, Wang, Frank Z. (2015) Spark on Entropy: A Reliable & Efficient Scheduler for Low-latency Parallel Jobs in Heterogeneous Cloud. In: Local Computer Networks Conference Workshops (LCN Workshops), 2015 IEEE 40th. . IEEE E-ISBN 978-1-4673-6773-8. (doi:10.1109/LCNW.2015.7365918)


In heterogeneous cloud, the provision of quality of service (QoS) guarantees for on-line parallel analysis jobs is much more challenging than off-line ones, mainly due to the many involved parameters, unstable resource performance, various job pattern and dynamic query workload. In this paper we propose an entropy-based scheduling strategy for running the on-line parallel analysis as a service more reliable and efficient, and implement the proposed idea in Spark. Entropy, as a measure of the degree of disorder in a system, is an indicator of a system’s tendency to progress out of order and into a chaotic condition, and it can thus serve to measure a cloud resource’s reliability for jobs scheduling. The key idea of our Entropy Scheduler is to construct the new resource entropy metric and schedule tasks according to the resources ranking with the help of the new metric so as to provide QoS guarantees for on-line Spark analysis jobs. Experiments demonstrate that our approach significantly reduces the average query response time by 15% - 20% and standard deviation by 30% - 45% compare with the native Fair Scheduler in Spark.

Item Type: Conference or workshop item (Paper)
DOI/Identification number: 10.1109/LCNW.2015.7365918
Uncontrolled keywords: Entropy Scheduler, Reliable, Low-latency, Spark, Jobs Scheduling, Heterogeneous Cloud, Cloud Computing
Subjects: Q Science > QA Mathematics (inc Computing science)
Divisions: Faculties > Sciences > School of Computing
Faculties > Sciences > School of Computing > Data Science
Depositing User: Frank Wang
Date Deposited: 02 Nov 2015 19:40 UTC
Last Modified: 29 May 2019 16:14 UTC
Resource URI: (The current URI for this page, for reference purposes)
  • Depositors only (login required):


Downloads per month over past year