Commit e5544aa9 authored by Daniel Seybold's avatar Daniel Seybold
Browse files

added new pages directly in repo

parent a3115f84
# Mowgli: Getting Started!
This guide describes how to setup and use the framework *Mowgli: Finding YourWay in the DBMS Jungle* which will be presented at the International Conference on Performance Engineering (ICPE) 2019.
This guide describes how to setup and use the framework *Mowgli: Finding YourWay in the DBMS Jungle* which was presented at the International Conference on Performance Engineering (ICPE) 2019.
The Mowgli framework enables the **performance**, **scalability**, **elasticity** and **availability** evaluation of distributed DBMS across cloud providers and cloud resources.
In the following, the required steps to setup, configure and use the Mowgli Framework are described.
**Requirement:** An account for an OpenStack-based cloud or the AWS EC2 cloud.
This guide consists of the following steps:
1. Setup Mowgli
2. Prepare Cloud in Mowgli framework
2.1. OpenStack
2.2. Amazon AWS EC2
3. Run Evaluation
## 1. Setup Mowgli
The Mowgli components are encapsulated as Docker containers and bundle via Docker-Compose.
The docker-compose contains all required components that are highlighted in blue.
![mowgli-setup](img/architecture_getting_started.png)
While the Mowgli framework can be run locally or on a VM, the Workload-API needs
to run on a separate VM within the selected cloud (to use the private network
and to ensure isolated workload creation). The default setup includes the Workload-API and assumes that the Mowgli framework is deployed in the same cloud and network that the evaluations are targeting. Additional Workload-APIs can added if required by following the [Workload-API installation guide](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Deploy-Workload-API) .
#### Install and start Mowgli Framework
As the Mowgli Framework comprises multiple dockerized components, Mowgli is provided as docker-compose bundle.
Though, the Mowgli framework can be deployed locally if the host is publicly accessible (unique IP and required ports open) and the Docker and Docker-Compose are installed
Yet, we recommend to create a dedicated VM.
The IP of the host of the Mowgli framework is denoted in the following as *MOWGLI_IP*.
| property | required |
| --- | --- |
| image / operating system | Ubuntu 16.04 |
| CPU cores | 4 |
| memory | 4GB |
| disk | 20GB |
| software | Docker, Docker-Compose |
| open ports | 3389 - RMI for the DBMS deployment via Cloudiator |
|| 4001 - ETCD to store Cloudiator deployment metadata |
|| 7001 - ETCD to store Cloudiator deployment metadata |
|| 8080 - the php-admin interface of the cloudiator dbms |
|| 8181 - the endpoint of the Workload-API |
|| 8086 - the REST interface of InfluxDB to store the monitoring data |
|| 8282 - the basic web interface of Mowgli framework |
|| 8888 - Chronograf, a web interface for the InfluxDB |
|| 9000 - the REST interface of Cloudiator |
|| 9001 - Portainer: a simple management UI for the docker containers |
|| 9091 - the basic web interface of dbms-catalogue |
Execute the following steps to deploy the Mowgli framework:
**Task 1**: install [docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) on your host system
**Task 2:** you have to change to root user: ```sudo -s```
**Task 3:** create the folder to store the evaluation results and plots: ```sudo mkdir -p /opt/evaluation-results/```
**Task 4:** create the folder for the Cloudiator data: ```sudo mkdir -p /var/lib/cloudiator/colosseum-store/```
**Task 5:** checkout the repository which includes the docker-compose.yml: ```git clone https://omi-gitlab.e-technik.uni-ulm.de/mowgli/docker.git```
**Task 6:** create the .env file based on the env-template: ```cd docker && cp env-template .env```
**Task 7:** add the public IP of the host the .env
**Task 8 (optional):** In case the host is running in an enterprise network and requires certificates please check the [wiki](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Add-Certificates)
**Task 9:** start the Mowgli services via: ```sudo docker-compose up -d```
**Task 10:** open your browser and check if the Mowgli framework started successfully at: `http://MOWGLI_HOST:8282` or `http://MOWGLI_HOST:8282/explorer`
If you can access the web interface you are nearly :smiley:
If not check the [troubleshooting](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Troubleshooting) section :scream:
In addition Mowgli comes with the [Portainer](https://www.portainer.io/) tool to provide a graphical UI for the Docker containers.
Portainer is accessible via http://MOWGLI_HOST:9001 with the default user *admin* and password *mowgli19*
## Prepare the target Cloud in the Mowgli framework
In the following, we will prepare Mowgli to use your Cloud. For this, we provide
two options:
1. Using the web interface at ```http://MOWGLI_HOST_IP:8282/``` or nicer web interface at `http://MOWGLI_HOST:8282/explorer`
2. Using your shell to directly send the REST calls
If you choose to use the shell scripts, you need to install curl, gridsite-clients, and jq[^3]. For this use: ```sudo apt-get update && sudo apt-get install -y curl gridsite-clients jq```
Mowgli enables the portable execution of evaluation scenarios over arbitrary OpenStack clouds and AWS EC2 by abstracting most of the technical details.
Yet, there a few **requirements** due to usage of [Cloudiator](http://cloudiator.org/) as deployment engine:
- the cloud needs to offer an Ubuntu 16 image that has `wget` installed (which is by default)
- the image name has to contain `ubuntu` and `16 ` to be selected for the deployment of the DBMS
- the default user of the image is `ubuntu`
- if your cloud does not provide an Ubuntu 16 image you can download one [here](https://cloud-images.ubuntu.com/xenial/) and upload it to your cloud
- Cloudiator can handle the security group management but it is required to allow all traffic within the private network to avoid connection issues. The required steps are explained in the following for OpenStack and EC2
In the following, the steps to register a cloud within Mowgli are explained.
**Note**: While your actual cloud credentials are required, they will never be exposed
via the API (which currently also means invalid credentials require wiping the
state of Cloudiator, see [troubleshooting](#### Reset Cloudiator State)!
### OpenStack
As first, we have to register a OpenStack cloud into the Mowgli framework. In order to do so, please make sure:
* if you have one or more private networks, get the ID of the preferred network
* volumes are not supported, flavors with root disks (>=70GB required for the scenarios described in the paper ) are required
* you are using either OpenStack with **Keystone** V2 or OpenStack V3
* get the **Keystone (identity)** endpoint URL of your OpenStack API, e.g. https://OPENSTACK_API:PORT/v3
* create the two security groups `mowgli-default` and `jclouds-mowgli`. The Mowgli framework will attach these security groups each spawned DBMS node (depending on the target cloud)
* `mowgli-default` and `jclouds-mowgli` must allow all traffic within the private network. In addition the ports `22,1099,4369,33033` need to be open to the public.
* See below an OpenStack example:
![security](img/security.png)
* In case you want to use public IPs instead you need to allow *all* traffic or specify the required ports for each DBMS manually. Yet, we recommend to use only private IPs for realistic evaluation scenarios.
#### OpenStack V2
##### Web interface
* Go to http://MOWGLI_HOST_IP:8282/#/default/templateCloudPost
* Click on *Try Out*.
* Enter the following parameters:
```
api: OPENSTACK
apiVersion: V2
apiEndpoint: https://OPENSTACK_API:PORT/v2.0
user: PROJECT_NAME:USER_NAME
secret: xxxx
```
* If you have more than one network you need to set the preferred network in the body:
```
[
{
"configKey": "NETWORK",
"configValue": "OPENSTACK_NETWORK_ID"
}
]
```
* Otherwise pass an empty array in the body:
```
[]
```
* Click on *Execute*.
* The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "openstack-19-12-2018_09:46:36"
}
```
##### Shell
* Execute the following in your shell (change the first 5 lines according to your credentials):
```
OS_USER=PROJECT_NAME:USER_NAME
OS_PASSWORD=SECRET
OS_API=https://YOUR_OS_CLOUD:5000/v2.0/
OS_VERSION=V2
OS_NETWORK=YOUR_NETWORK_ID # leave empty if you do not have multiple networks
MOWGLI_HOST_IP=x.x.x.x
if [ $OS_NETWORK ]; then OS_NETWORK="{ \"configKey\": \"NETWORK\", \"configValue\": \"$(urlencode $OS_NETWORK)\" }"; else OS_NETWORK=""; fi
curl -X POST "http://$MOWGLI_HOST_IP:8282/v1/template/cloud?api=OPENSTACK&apiVersion=$OS_VERSION&apiEndpoint=$(urlencode $OS_API)&user=$(urlencode $OS_USER)&secret=$(urlencode $OS_PASSWORD)" -H "accept: application/json" -H "Content-Type: application/json" -d "[$OS_NETWORK]" | jq
```
* The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "openstack-19-12-2018_09:46:36"
}
```
#### OpenStack V3
##### Web interface
* Go to http://MOWGLI_HOST_IP:8282/#/default/templateCloudPost
* Click on *Try Out*.
* Enter the following parameters:
```
api: OPENSTACK
apiVersion: V3
apiEndpoint: https://OPENSTACK_API:PORT/v3
user: USER_DOMAIN:PROJECT_NAME/PROJECT_ID:USER_NAME
secret: xxxx
```
* all parameters can be found in the **OpenStack RC File** that can be downloaded in the OpenStack dashboard under **API Access**
Topics:
* USER_DOMAIN will be most likely `default`
1. Requirements and Feature Overview
* If you have more than one network you need to set the preferred network in the body:
2. [Setup Mowgli Framework](setup/README.md)
```
[
{
"configKey": "NETWORK",
"configValue": "OPENSTACK_NETWORK_ID"
}
]
```
3. [Usage](usage/README.md)
* Otherwise pass an empty array in the body:
```
[]
```
3.1 Performance Scenarios
* Click on *Execute*.
3.2 Scalability Scenarios
* The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "openstack-19-12-2018_09:46:36"
}
```
3.3 Elasticity Scenarios
##### Shell
* Execute the following in your shell (change the first 5 lines according to your credentials):
```
OS_USER=USER_DOMAIN:PROJECT_ID:USER_NAME (USER_DOMAIN is probabely default)
OS_PASSWORD=SECRET
OS_API=https://YOUR_OS_CLOUD:5000/v3/
OS_VERSION=V3
OS_NETWORK=YOUR_NETWORK_ID # leave empty if you do not have multiple networks
MOWGLI_HOST_IP=x.x.x.x
if [ $OS_NETWORK ]; then OS_NETWORK="{ \"configKey\": \"NETWORK\", \"configValue\": \"$(urlencode $OS_NETWORK)\" }"; else OS_NETWORK=""; fi
curl -X POST "http://$MOWGLI_HOST_IP:8282/v1/template/cloud?api=OPENSTACK&apiVersion=$OS_VERSION&apiEndpoint=$(urlencode $OS_API)&user=$(urlencode $OS_USER)&secret=$(urlencode $OS_PASSWORD)" -H "accept: application/json" -H "Content-Type: application/json" -d "[$OS_NETWORK]" | jq
```
* The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "openstack-19-12-2018_09:46:36"
}
```
### Amazon AWS EC2
Currently Cloudiator does not support EBS volumes on EC2. This requires that the creation of a custom flavor with a predefined EBS volume following these steps:
1. Select the Ubuntu Server 16.04 as base image
![ec2 step1](img/step1.png)
2. Select a flavor (t2.medium recommended as this was used in the paper)
![ec2 step2](img/step2.png)
3. Use the default settings
![ec2 step3](img/step3.png)
4. Specify the disk size and type for the flavor and launch the instance
![ec2 step4](img/step4.png)
5. Create an image of the running instance which will be used as custom flavor
![ec2 step5](img/step5.png)
6. Apply valid name to the image that includes `ubuntu` and `16` so it will be discovered by Mowgli
![ec2 step6](img/step7b.png)
7. Create the custom image and keep the ami-ID which will be required later
![ec2 step7](img/step8.png)
To allow the communication between all nodes in your private network extend the default security group to allow all traffic within the default security group.
![ec2 step8](img/step9.png)
Register your cloud via the http://MOWGLI_HOST_IP:8282/#/default/templateCloudPost
with the settings:
```
api: EC2
apiVersion: DEFAULT
apiEndpoint:
user: user-id
secret: api-key
```
If you have more than one network you need to set the preferred network in the body as `configValue`. In addition you need to set the image filter to only get the previously created image (otherwise Cloudiator will fetch all public images of EC2!)
```
[
{
"configKey": "NETWORK",
"configValue": "EC2_NETWORK_ID"
},
{
"configKey": "AMIQUERY",
"configValue": "ami-093f8fa74f6c6b180"
},
{
"configKey": "AMICCQUERY",
"configValue": "ami-093f8fa74f6c6b180"
}
]
```
The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "ec2-19-12-2018_09:46:36"
}
```
## Run Evaluation
After the successful registration of the cloud, Cloudiator will collect the cloud resource offerings of the cloud provider.
Depending on the number of cloud resource offers this might take some minutes, so time for a coffee ;-)
Mowgli will compose the cloud resource offerings into VM templates that are required by each evaluation scenario.
Now you can query the Mowgli framework for appropriate VM templates as described [here](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Get-VM-Templates)
### Start evaluation
After getting the VM templates you are ready to start the evaluations :cloud: :hourglass: :trophy:
Mowgli supports four types of evaluation scenarios:
[Performance](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Performance-Evaluation)
[Scalability](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Scalability-Evaluation)
Elasticity
Availability
Please check the respective scenario pages for further details about the execution, the supported DBMS and workloads.
3.4 Availability Scenarios
In the following, the required steps to setup, configure and use the Mowgli Framework are described.
**Requirement:** An account for an OpenStack-based cloud or the AWS EC2 cloud.
## Evaluation Results
This guide consists of the following steps:
All evaluation results are stored to the file system of the host that runs the Mowgli Framework in the following structure
1. Setup Mowgli
2. Prepare Cloud in Mowgli framework
2.1. OpenStack
2.2. Amazon AWS EC2
3. Run Evaluation
```
opt
|_evaluation-results
|_SCENARIO
|_CLOUD
|_DBMS
|_CONFIG
|_RUN_X
|_data # contains raw evaluation data
|_monitoring # contains system usage plots
|_specs # contains the applied templates
|_taskLogs # additional logs
|_timeseries # throughput plot of the evaluation run
|_plots # contains aggregated evaluation data
```
By default the plots for throughput and latency are generated under the timeseries folder.
In addition system metric plots of the Workload-API instances and the DBMS nodes are available under the monitoring folder.
## References
......
## 1. Setup Mowgli
The Mowgli components are encapsulated as Docker containers and bundle via Docker-Compose.
The docker-compose contains all required components that are highlighted in blue.
![mowgli-setup](../misc/architecture_getting_started.png)
While the Mowgli framework can be run locally or on a VM, the Workload-API needs
to run on a separate VM within the selected cloud (to use the private network
and to ensure isolated workload creation). The default setup includes the Workload-API and assumes that the Mowgli framework is deployed in the same cloud and network that the evaluations are targeting. Additional Workload-APIs can added if required by following the [Workload-API installation guide](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Deploy-Workload-API) .
#### Install and start Mowgli Framework
As the Mowgli Framework comprises multiple dockerized components, Mowgli is provided as docker-compose bundle.
Though, the Mowgli framework can be deployed locally if the host is publicly accessible (unique IP and required ports open) and the Docker and Docker-Compose are installed
Yet, we recommend to create a dedicated VM.
The IP of the host of the Mowgli framework is denoted in the following as *MOWGLI_IP*.
| property | required |
| ------------------------ | ------------------------------------------------------------ |
| image / operating system | Ubuntu 16.04 |
| CPU cores | 4 |
| memory | 4GB |
| disk | 20GB |
| software | Docker, Docker-Compose |
| open ports | 3389 - RMI for the DBMS deployment via Cloudiator |
| | 4001 - ETCD to store Cloudiator deployment metadata |
| | 7001 - ETCD to store Cloudiator deployment metadata |
| | 8080 - the php-admin interface of the cloudiator dbms |
| | 8181 - the endpoint of the Workload-API |
| | 8086 - the REST interface of InfluxDB to store the monitoring data |
| | 8282 - the basic web interface of Mowgli framework |
| | 8888 - Chronograf, a web interface for the InfluxDB |
| | 9000 - the REST interface of Cloudiator |
| | 9001 - Portainer: a simple management UI for the docker containers |
| | 9091 - the basic web interface of dbms-catalogue |
Execute the following steps to deploy the Mowgli framework:
**Task 1**: install [docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) on your host system
**Task 2:** you have to change to root user: ```sudo -s```
**Task 3:** create the folder to store the evaluation results and plots: ```sudo mkdir -p /opt/evaluation-results/```
**Task 4:** create the folder for the Cloudiator data: ```sudo mkdir -p /var/lib/cloudiator/cloudiator-store/```
**Task 5:** checkout the repository which includes the docker-compose.yml: ```git clone https://omi-gitlab.e-technik.uni-ulm.de/mowgli/docker.git```
**Task 6:** create the .env file based on the env-template: ```cd docker && cp env-template .env```
**Task 7:** add the public IP of the host the .env
**Task 8 (optional):** In case the host is running in an enterprise network and requires certificates please check the [wiki](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Add-Certificates)
**Task 9:** start the Mowgli services via: ```sudo docker-compose up -d```
**Task 10:** open your browser and check if the Mowgli framework started successfully at: `http://MOWGLI_HOST:8282` or `http://MOWGLI_HOST:8282/explorer`
If you can access the web interface you are nearly there :smiley:
If not check the [troubleshooting](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Troubleshooting) section :scream:
In addition Mowgli comes with the [Portainer](https://www.portainer.io/) tool to provide a graphical UI for the Docker containers.
Portainer is accessible via http://MOWGLI_HOST:9001 with the default user *admin* and password *mowgli19*
## Prepare the target Cloud in the Mowgli framework
In the following, we will prepare Mowgli to use your Cloud. For this, we provide
two options:
1. Using the default web interface at ```http://MOWGLI_HOST_IP:8282/```
2. (recommended) Using the more responsive web interface at `http://MOWGLI_HOST:8282/explorer`
Mowgli enables the portable execution of evaluation scenarios over arbitrary OpenStack clouds and AWS EC2 by abstracting most of the technical details.
Yet, there a few **requirements** due to usage of [Cloudiator](http://cloudiator.org/) as deployment engine:
- the cloud needs to offer an Ubuntu 16 image that has `wget` installed (which is by default)
- the image name has to contain `ubuntu` and `16 ` to be selected for the deployment of the DBMS
- the default user of the image is `ubuntu`
- if your cloud does not provide an Ubuntu 16 image you can download one [here](https://cloud-images.ubuntu.com/xenial/) and upload it to your cloud
- Cloudiator can handle the security group management but it is required to allow all traffic within the private network to avoid connection issues. The required steps are explained in the following for OpenStack and EC2
In the following, the steps to register a cloud within Mowgli are explained.
**Note**: While your actual cloud credentials are required, they will never be exposed
via the API (which currently also means invalid credentials require wiping the
state of Cloudiator, see [../troubleshooting.README.md](#### Reset Cloudiator State)!
### OpenStack
As first, we have to register a OpenStack cloud into the Mowgli framework. In order to do so, please make sure:
* if you have one or more private networks, get the ID of the preferred network
* volumes are not supported, flavors with root disks are required (>=70GB required for the scenarios described in the Mowgli paper )
* check if you are using either OpenStack with **Keystone** V2 or OpenStack V3
* get the **Keystone (identity)** endpoint URL of your OpenStack API, e.g. https://OPENSTACK_API:PORT/v3
* create the two security groups `mowgli-default` and `jclouds-mowgli`. The Mowgli framework will attach these security groups each spawned DBMS node (depending on the target cloud)
* `mowgli-default` and `jclouds-mowgli` must allow all traffic within the private network. In addition the ports `22,1099,4369,33033` need to be open to the public.
* See below an OpenStack example:
![security](../misc/security.png)
* In case you want to use public IPs instead you need to allow *all* traffic or specify the required ports for each DBMS manually. Yet, we recommend to use only private IPs for realistic evaluation scenarios.
* all parameters can be found in the **OpenStack RC File** that can be downloaded in the OpenStack dashboard under **API Access**
#### OpenStack V2
##### Web interface
* Go to http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost
* Enter the following parameters:
```
api: OPENSTACK
apiVersion: V2
user: PROJECT_NAME:USER_NAME
secret: xxxx
apiEndpoint: https://OPENSTACK_API:PORT/v2.0
```
* If you have more than one network you need to set the preferred network in the body:
```
[
{
"configKey": "NETWORK",
"configValue": "OPENSTACK_NETWORK_ID"
}
]
```
* Otherwise pass an empty array in the body:
```
[]
```
* Click on *Try*.
* The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "openstack-19-12-2018_09:46:36"
}
```
#### OpenStack V3
##### Web interface
* Go to http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost
* Enter the following parameters:
```
api: OPENSTACK
apiVersion: V3
user: USER_DOMAIN:PROJECT_NAME/PROJECT_ID:USER_NAME
secret: xxxx
apiEndpoint: https://OPENSTACK_API:PORT/v3
```
* USER_DOMAIN will be most likely `default`
* If you have more than one network you need to set the preferred network in the body:
```
[
{
"configKey": "NETWORK",
"configValue": "OPENSTACK_NETWORK_ID"
}
]
```
* Otherwise pass an empty array in the body:
```
[]
```
* Click on *Execute*.
* The following message indicates the successful registration of the cloud:
```
{
"id": 1,
"name": "openstack-19-12-2018_09:46:36"
}
```
### Amazon AWS EC2
Currently Cloudiator does not support EBS volumes on EC2. This requires that the creation of a custom flavor with a predefined EBS volume following these steps:
1. Select the Ubuntu Server 16.04 as base image