Skip to main content

A cloud-agnostic queuing system to support the implementation of deadline-based application execution policies

Kiss, Tamas, DesLauriers, James, Gesmier, Gregoire, Terstyanszky, Gabor, Pierantoni, Gabriele, Oun, Osama Abu, Taylor, Simon J.E., Anagnostou, Anastasia, Kovacs, Jozsef (2019) A cloud-agnostic queuing system to support the implementation of deadline-based application execution policies. Future Generation Computer Systems, 101 . pp. 99-111. ISSN 0167-739X. (doi:10.1016/j.future.2019.05.062) (KAR id:75315)

PDF Publisher pdf
Language: English


Download (2MB) Preview
[thumbnail of 1-s2.0-S0167739X19303814-main.pdf]
Preview
This file may not be suitable for users of assistive technology.
Request an accessible format
Official URL
https://doi.org/10.1016/j.future.2019.05.062

Abstract

There are many scientific and commercial applications that require the execution of a large number of independent jobs resulting in significant overall execution time. Therefore, such applications typically require distributed computing infrastructures and science gateways to run efficiently and to be easily accessible for end-users. Optimising the execution of such applications in a cloud computing environment by keeping resource utilisation at minimum but still completing the experiment by a set deadline has paramount importance. As container-based technologies are becoming more widespread, support for job-queuing and auto-scaling in such environments is becoming important. Current container management technologies, such as Docker Swarm or Kubernetes, while provide auto-scaling based on resource consumption, do not support job queuing and deadline-based execution policies directly. This paper presents JQueuer, a cloud-agnostic queuing system that supports the scheduling of a large number of jobs in containerised cloud environments. The paper also demonstrates how JQueuer, when integrated with a cloud application-level orchestrator and auto-scaling framework, called MiCADO, can be used to implement deadline-based execution policies. This novel technical solution provides an important step towards the cost-optimisation of batch processing and job submission applications. In order to test and prove the effectiveness of the solution, the paper presents experimental results when executing an agent-based simulation application using the open source REPAST simulation framework.

Item Type: Article
DOI/Identification number: 10.1016/j.future.2019.05.062
Uncontrolled keywords: Cloud computing, Container technologies, Deadline-based auto-scaling, JQueuer, MiCADO, Agent-based simulation
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Depositing User: Osama Abu Oun
Date Deposited: 12 Jul 2019 09:21 UTC
Last Modified: 16 Feb 2021 14:05 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/75315 (The current URI for this page, for reference purposes)
Oun, Osama Abu: https://orcid.org/0000-0003-4884-8011
  • Depositors only (login required):

Downloads

Downloads per month over past year