CEEM: A Practical Methodology for Cloud Services Evaluation

Abstract—Given an increasing number of Cloud services
available in the market, evaluating candidate Cloud services
is crucial and beneficial for both service customers (e.g. costbenefit analysis) and providers (e.g. direction of improvement).
When it comes to performing any evaluation, a suitable
methodology is inevitably required to direct experimental
implementations. Nevertheless, there is still a lack of a sound methodology to guide the evaluation of Cloud services. By borrowing the lessons from evaluation of traditional computing systems, referring to the guidelines for Design of Experiments (DOE), and summarizing the existing experiences of real experimental studies, we proposed a generic Cloud Evaluation Experiment Methodology (CEEM) for Cloud services evaluation.
Furthermore, we have established a pre-experimental
knowledge base and specified corresponding suggestions to
make this methodology more practical in the Cloud Computing
domain. Through evaluating the Google AppEngine Python
runtime as a preliminary validation, we show that Cloud
evaluators may achieve more rational and convincing experimental results and conclusions following such an evaluation methodology.