Skip to main content
Kent Academic Repository

Spark on Entropy: A Reliable & Efficient Scheduler for Low-latency Parallel Jobs in Heterogeneous Cloud

Chen, Huankai, Wang, Frank Z. (2015) Spark on Entropy: A Reliable & Efficient Scheduler for Low-latency Parallel Jobs in Heterogeneous Cloud. In: Local Computer Networks Conference Workshops (LCN Workshops), 2015 IEEE 40th. . IEEE E-ISBN 978-1-4673-6773-8. (doi:10.1109/LCNW.2015.7365918) (KAR id:51357)


In heterogeneous cloud, the provision of quality of

service (QoS) guarantees for on-line parallel analysis jobs is much

more challenging than off-line ones, mainly due to the many

involved parameters, unstable resource performance, various job

pattern and dynamic query workload. In this paper we propose

an entropy-based scheduling strategy for running the on-line

parallel analysis as a service more reliable and efficient, and

implement the proposed idea in Spark.

Entropy, as a measure of the degree of disorder in a system,

is an indicator of a system’s tendency to progress out of order

and into a chaotic condition, and it can thus serve to measure a

cloud resource’s reliability for jobs scheduling. The key idea of

our Entropy Scheduler is to construct the new resource entropy

metric and schedule tasks according to the resources ranking with

the help of the new metric so as to provide QoS guarantees for

on-line Spark analysis jobs. Experiments demonstrate that our

approach significantly reduces the average query response time

by 15% - 20% and standard deviation by 30% - 45% compare

with the native Fair Scheduler in Spark.

Item Type: Conference or workshop item (Paper)
DOI/Identification number: 10.1109/LCNW.2015.7365918
Uncontrolled keywords: Entropy Scheduler, Reliable, Low-latency, Spark, Jobs Scheduling, Heterogeneous Cloud, Cloud Computing
Subjects: Q Science > QA Mathematics (inc Computing science)
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Depositing User: Frank Wang
Date Deposited: 02 Nov 2015 19:40 UTC
Last Modified: 08 Dec 2022 22:46 UTC
Resource URI: (The current URI for this page, for reference purposes)

University of Kent Author Information

Chen, Huankai.

Creator's ORCID:
CReDIT Contributor Roles:

Wang, Frank Z..

Creator's ORCID:
CReDIT Contributor Roles:
  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.