Chen, Huankai, Wang, Frank Z. (2015) Spark on Entropy: A Reliable & Efficient Scheduler for Low-latency Parallel Jobs in Heterogeneous Cloud. In: Local Computer Networks Conference Workshops (LCN Workshops), 2015 IEEE 40th. . IEEE E-ISBN 978-1-4673-6773-8. (doi:10.1109/LCNW.2015.7365918) (KAR id:51357)
PDF
Language: English |
|
Download this file (PDF/402kB) |
|
Request a format suitable for use with assistive technology e.g. a screenreader | |
Official URL: https://doi.org/10.1109/LCNW.2015.7365918 |
Abstract
In heterogeneous cloud, the provision of quality of
service (QoS) guarantees for on-line parallel analysis jobs is much
more challenging than off-line ones, mainly due to the many
involved parameters, unstable resource performance, various job
pattern and dynamic query workload. In this paper we propose
an entropy-based scheduling strategy for running the on-line
parallel analysis as a service more reliable and efficient, and
implement the proposed idea in Spark.
Entropy, as a measure of the degree of disorder in a system,
is an indicator of a system’s tendency to progress out of order
and into a chaotic condition, and it can thus serve to measure a
cloud resource’s reliability for jobs scheduling. The key idea of
our Entropy Scheduler is to construct the new resource entropy
metric and schedule tasks according to the resources ranking with
the help of the new metric so as to provide QoS guarantees for
on-line Spark analysis jobs. Experiments demonstrate that our
approach significantly reduces the average query response time
by 15% - 20% and standard deviation by 30% - 45% compare
with the native Fair Scheduler in Spark.
Item Type: | Conference or workshop item (Paper) |
---|---|
DOI/Identification number: | 10.1109/LCNW.2015.7365918 |
Uncontrolled keywords: | Entropy Scheduler, Reliable, Low-latency, Spark, Jobs Scheduling, Heterogeneous Cloud, Cloud Computing |
Subjects: | Q Science > QA Mathematics (inc Computing science) |
Divisions: | Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing |
Depositing User: | Frank Wang |
Date Deposited: | 02 Nov 2015 19:40 UTC |
Last Modified: | 05 Nov 2024 10:37 UTC |
Resource URI: | https://kar.kent.ac.uk/id/eprint/51357 (The current URI for this page, for reference purposes) |
- Link to SensusAccess
- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV
- Depositors only (login required):