Commit 37eb989c authored by Daniel Seybold's avatar Daniel Seybold
Browse files

extended analytics parts

parent 4a92acd8
......@@ -6,9 +6,7 @@ The docker-compose contains all required components that are highlighted in blue
![mowgli-setup](../misc/mowgli_architecture_spec_2.png)
While the Mowgli framework can be run locally or on a VM, the Workload-API needs
to run on a separate VM within the selected cloud (to use the private network
and to ensure isolated workload creation). The default setup includes the Workload-API and assumes that the Mowgli framework is deployed in the same cloud and network that the evaluations are targeting. Additional Workload-APIs can added if required by following the [Workload-API installation guide](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Deploy-Workload-API) .
While the Mowgli framework can be run locally or on a VM, the Workload-API needs to run on a separate VM within the selected cloud (to use the private network and to ensure isolated workload creation). The default setup includes the Workload-API and assumes that the Mowgli framework is deployed in the same cloud and network that the evaluations are targeting. Additional Workload-APIs can added if required by following the [Workload-API installation guide](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Deploy-Workload-API) .
In the following, the required steps to setup, configure and use the Mowgli Framework are described.
......@@ -38,7 +36,7 @@ The IP of the host of the Mowgli framework is denoted in the following as *MOWGL
| property | required |
| ------------------------ | ------------------------------------------------------------ |
| image / operating system | Ubuntu 16.04 |
| image / operating system | Ubuntu 18.04 (recommended), any OS with docker and docker-compose should work |
| CPU cores | 4 |
| memory | 4GB |
| disk | 20GB |
......@@ -88,8 +86,8 @@ two options:
Mowgli enables the portable execution of evaluation scenarios over arbitrary OpenStack clouds and AWS EC2 by abstracting most of the technical details.
Yet, there a few **requirements** due to usage of [Cloudiator](http://cloudiator.org/) as deployment engine:
- the cloud needs to offer an Ubuntu 16 image that has `wget` installed (which is by default)
- the image name has to contain `ubuntu` and `16 ` to be selected for the deployment of the DBMS
- the cloud needs to offer an **Ubuntu 16 (recommended, Ubuntu 18 should also work)** image that has `wget` installed (which is by default)
- the image name has to contain `ubuntu` and `16 ` (or `18`) to be selected for the deployment of the DBMS
- the default user of the image is `ubuntu`
- if your cloud does not provide an Ubuntu 16 image you can download one [here](https://cloud-images.ubuntu.com/xenial/) and upload it to your cloud
- Cloudiator can handle the security group management but it is required to allow all traffic within the private network to avoid connection issues. The required steps are explained in the following for OpenStack and EC2
......
# Usage
## Run Evaluation
After the successful registration of the cloud, Cloudiator will collect the cloud resource offerings of the cloud provider.
Depending on the number of cloud resource offers this might take some minutes, so time for a coffee :coffee:
Mowgli will compose the cloud resource offerings into VM templates that are required by each evaluation scenario.
Now you can query the Mowgli framework for appropriate VM templates as described [here](Get-VM-Templates.md)
### Start evaluation
After getting the VM templates you are ready to start the evaluations :cloud: :hourglass: :trophy:
Mowgli supports four types of evaluation scenarios:
[Performance](Performance-Evaluation.md)
[Scalability](Scalability-Evaluation.md)
[Elasticity](Elasticity-Evaluation.md)
[Availability](Availability-Evaluation.md)
Please check the respective scenario pages for further details about the execution, the supported DBMS and workloads.
## Evaluation Results
All evaluation results are stored to the file system of the host that runs the Mowgli Framework in the following structure
```
opt
|_evaluation-results
|_SCENARIO
|_CLOUD
|_DBMS
|_CONFIG
|_RUN_X
|_data # contains raw evaluation data
|_monitoring # contains system usage plots
|_specs # contains the applied templates
|_taskLogs # additional logs
|_timeseries # throughput plot of the evaluation run
|_plots # contains aggregated evaluation data over all runs (manual processing steps required)
```
More details can be found in the respective sections.
\ No newline at end of file
......@@ -6,18 +6,13 @@ Performance scenarios deploy a single node DBMS on the specified cloud resource
The resulting metrics are throughput and latency.
------
For a list of supported DBMS and Benchmarks please check the [feature overview](../features/README.md)
#### Supported DBMS
- Cassandra
- CockroachDB
- Couchbase
- Elasticsearch
- MongoDB (beta)
- Yugabyte (beta)
More details on the concepts can be found in [Mowgli: Finding Your Way in the DBMS Jungle](https://dl.acm.org/doi/10.1145/3297663.3310303)
------
......@@ -112,33 +107,61 @@ opt
|_specs # contains the applied templates
|_taskLogs # additional logs
|_timeseries # throughput plot of the evaluation run
|_plots # contains aggregated evaluation data over all runs (manual processing steps required)
|_plots # contains aggregated evaluation data over all runs (manual processing steps required)
```
##### Evaluation Scenario Specs
#### DATA
The `data` folder contains the raw evaluation results of the selected workload.
**YCSB**
The `data` folder contains the raw evaluation results of the load phase in the load.txt and the CRUD (or transaction phase in YCSB context) in the transaction phase.
**TPC-C**
The `data` folder contains the raw evaluation results.
Currently no advanced plotting support is available.
#### MONITORING
Under the `specs` folder you will find the applied DBMS and cloud resources under the `dbmsSpec.json` and the applied workload specs as JSON files.
The `monitoring` folder contains raw system metrics of the DBMS and Workload-API nodes. In addition, these metrics are automatically plotted as timeseries as shown in the exemplary plot.
![CPU Load](../misc/cpuLoad.png)
#### SPECS
The `specs` folder contains evaluation scenario specs the that have been applied. You will find the applied DBMS and cloud resources under the `dbmsSpec.json` and the applied workload specs, i.e. `0_loadSpec.json` or `0_transactionSpec.json`, as JSON files.
In addition you can will find the `resourceMapping.json` that contains a mapping of the applied abstract VM template to the concrete properties of the selected cloud provider.
##### Workload Metrics
#### TASKLOGS
The `data` folder contains the raw evaluation results of the selected workload
The `taskLogs` folder contains additional metadata as times series of the events that have been triggered during the evaluation, e.g. VM start-up times, deployment times.
**YCSB**
#### TIMESERIES
The `data` folder contains the raw evaluation results of the load phase in the load.txt and the CRUD (or transaction phase in YCSB context) in the transaction phase.
The `timeseries` folder contains plots for throughput and latency metrics as depicted in the following plot. This feature is currently only supported for YCSB.
By default the plots for throughput and latency are generated under the timeseries folder.
![](../misc/load-throughput-ts.png)
In addition system metric plots of the Workload-API instances and the DBMS nodes are available under the monitoring folder.
**TPC-C**
The `data` folder contains the raw evaluation results.
------
Currently no advanced plotting support is available.
#### Data Analytics
Besides the automated processing of the previously described steps, the Mowgli framework provides additional tools for advanced data processing that are currently not automated within the framework.
These tools comprise the aggregations and statically analysis of multiple runs or different evaluation scenarios.
In addition system metric plots of the Workload-API instances and the DBMS nodes are available under the monitoring folder.
\ No newline at end of file
The set of supportive data analytic script is described under the [analytics section](Data-Analytics.md).
\ No newline at end of file
......@@ -18,13 +18,13 @@ After getting the VM templates you are ready to start the evaluations :cloud: :h
Mowgli supports four types of evaluation scenarios:
[Performance](Performance-Evaluation.md) (v0.1)
[Performance](Performance-Evaluation.md)
[Scalability](Scalability-Evaluation.md) (v0.1)
[Scalability](Scalability-Evaluation.md)
[Elasticity](Elasticity-Evaluation.md) (v0.2)
[Elasticity](Elasticity-Evaluation.md)
[Availability](Availability-Evaluation.md) (v0.2)
[Availability](Availability-Evaluation.md)
Please check the respective scenario pages for further details about the execution, the supported DBMS and workloads.
......@@ -52,8 +52,4 @@ opt
```
The `data` folder contains the raw evaluation results of the selected workload
By default the plots for throughput and latency are generated under the timeseries folder (currently only supported for YCSB workload).
In addition system metric plots of the Workload-API instances and the DBMS nodes are available under the monitoring folder.
\ No newline at end of file
More details can be found in the respective sections.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment