Entropy-Based Resource Management in Complex Cloud Environment

Resource Management is an NP-complete problem, the complexity of which increases substantially in the Cloud environment. The complexity of cloud resource management can originate from many factors: the scale of the resources; the heterogeneity of the resource types and the interdependencies of these; as well as the variability, dynamicity, and unpredictability of resource run-time performance.

Complexity has many negative effects in relation to satisfying the Quality of Service (QoS) requirements of cloud applications, such as cost, performance, availability, and reliability. If an application cannot guarantee its QoS, it will be hard to populate. However, the vast majority of research efforts into cloud resource management implicitly assume the Cloud to be a simplifying technology and that the cloud resource's performance is determined and predictable. These incorrect assumptions may significantly affect the QoS of any cloud application developed under it, causing its resource management strategy to be less than robust.

In spite of there being extensive research into complexity issues in many diverse fields ranging from computational biology to decision making in economics, the study of complexity in cloud resource management systems is limited. In this thesis, I address the complexity problems of Cloud Resource Management Systems by introducing the use of Entropy Theory in relation to them.

The main contributions of this thesis are as follows:

1. A cloud simulation tool-kit, ComplexCloudSim, is implemented in order to help tackle the research question: what is the role of complexity in QoS-aware cloud resource management? The uncovering of Chaotic Behaviour in Cloud Resource Management Systems by using the Damage Spreading Analysis method.

2. The comprehensive definition of complexity in the Cloud Resource Management Systems; such can be primarily classified into two categories: Global System Complexity and Local Resource Complexity.

3. An Entropy Theory based resource management model is proposed for the purposes of identifying, measuring, analyzing and controlling (i.e., reducing and avoiding) complexity.

4. A Cellular Automata Entropy based methodology is proposed as a solution to the Cloud resource allocation problem; this methodology is capable of managing Global System Complexity.

6. Once the root cause of the complexity has been identified using the Local Activity Principle, a Resource Entropy Based Local Activity Ranking system can be proposed which solves the job scheduling problem by managing Local Resource Complexity. Finally, on this latter basis, I implement a system which I have termed an Entropy Scheduler within a popular real-world cloud analysis engine, Apache Spark. Experiments demonstrate that the new Entropy Scheduler significantly reduces the average query response time by 15% - 20% and standard deviation by 30% - 45% compare with the native Fair Scheduler for running CPU intensive applications in Apache Spark, when the Spark server is not overloaded.