## 1. Setup Mowgli The Mowgli components are encapsulated as Docker containers and bundle via Docker-Compose. The docker-compose contains all required components that are highlighted in blue. ![mowgli-setup](../misc/mowgli_architecture_spec_2.png) While the Mowgli framework can be run locally or on a VM, the Workload-API needs to run on a separate VM within the selected cloud (to use the private network and to ensure isolated workload creation). The default setup includes the Workload-API and assumes that the Mowgli framework is deployed in the same cloud and network that the evaluations are targeting. Additional Workload-APIs can added if required by following the [Workload-API installation guide](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Deploy-Workload-API) . In the following, the required steps to setup, configure and use the Mowgli Framework are described. **Requirement:** An account for an OpenStack-based cloud or the AWS EC2 cloud. This guide consists of the following steps: 1. Install Mowgli 2. Prepare and register the target cloud in Mowgli 2.1. OpenStack 2.2. Amazon AWS EC2 #### 1. Install and start Mowgli Framework As the Mowgli Framework comprises multiple dockerized components, Mowgli is provided as docker-compose bundle. Though, the Mowgli framework can be deployed locally if the host is publicly accessible (unique IP and required ports open) and the Docker and Docker-Compose are installed Yet, we recommend to create a dedicated VM. The IP of the host of the Mowgli framework is denoted in the following as *MOWGLI_IP*. | property | required | | ------------------------ | ------------------------------------------------------------ | | image / operating system | Ubuntu 18.04 (recommended), any Linux-based OS with docker and docker-compose installed will work | | CPU cores | 4 | | memory | 4GB | | disk | 20GB | | software | Docker, Docker-Compose | | open ports | 3389 - RMI for the DBMS deployment via Cloudiator | | | 4001 - ETCD to store Cloudiator deployment metadata | | | 7001 - ETCD to store Cloudiator deployment metadata | | | 8080 - the php-admin interface of the cloudiator dbms | | | 8181 - the endpoint of the Workload-API | | | 8086 - the REST interface of InfluxDB to store the monitoring data | | | 8282 - the basic web interface of Mowgli framework | | | 8888 - Chronograf, a web interface for the InfluxDB | | | 9000 - the REST interface of Cloudiator | | | 9001 - Portainer: a simple management UI for the docker containers | | | 9091 - the basic web interface of dbms-catalogue | Execute the following steps to deploy the Mowgli framework: **Task 1**: install [docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) on your host system **Task 2:** you have to change to root user: ```sudo -s``` **Task 3:** create the folder to store the evaluation results and plots: ```sudo mkdir -p /opt/evaluation-results/``` **Task 4:** create the folder for the Cloudiator data: ```sudo mkdir -p /var/lib/cloudiator/cloudiator-store/``` **Task 5:** checkout the repository which includes the docker-compose.yml: ```git clone -b 0.2 https://omi-gitlab.e-technik.uni-ulm.de/mowgli/docker.git mowgli``` **Task 6:** create the .env file based on the env-template: ```cd mowgli && cp env-template .env``` **Task 7:** add the **publicly accessible IP** of the host to the .env **Task 8 (optional):** In case the host is running in an enterprise network and requires certificates please check the [wiki](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Add-Certificates) **Task 9:** start the Mowgli services via: ```sudo docker-compose up -d``` **Task 10:** open your browser and check if the Mowgli framework started successfully at: `http://MOWGLI_HOST:8282` or `http://MOWGLI_HOST:8282/explorer` If you can access the web interface you are nearly there :smiley: If not check the [troubleshooting](../troubleshooting/README.md) section :scream: In addition Mowgli comes with the [Portainer](https://www.portainer.io/) tool to provide a graphical UI for the Docker containers. Portainer is accessible via http://MOWGLI_HOST:9001 with the default user *admin* and password *mowgli19* ## 2. Prepare and register the target Cloud in Mowgli In the following, we will prepare Mowgli to use your Cloud. For this, we provide two options: 1. Using the default web interface at ```http://MOWGLI_HOST_IP:8282/``` 2. (recommended) Using the more responsive web interface at `http://MOWGLI_HOST:8282/explorer` Mowgli enables the portable execution of evaluation scenarios over arbitrary OpenStack clouds and AWS EC2 by abstracting most of the technical details. Yet, there a few **requirements** due to usage of [Cloudiator](http://cloudiator.org/) as deployment engine: - the cloud needs to offer an **Ubuntu 16 (recommended, Ubuntu 18 should also work)** image that has `wget` installed (which is by default) - the image name has to contain `ubuntu` and `16 ` (or `18`) to be selected for the deployment of the DBMS - the default user of the image is `ubuntu` - if your cloud does not provide an Ubuntu 16 image you can download one [here](https://cloud-images.ubuntu.com/xenial/) and upload it to your cloud - Cloudiator can handle the security group management but it is required to allow all traffic within the private network to avoid connection issues. The required steps are explained in the following for OpenStack and EC2 In the following, the steps to register a cloud within Mowgli are explained. **Note**: While your actual cloud credentials are required, they will never be exposed via the API (which currently also means invalid credentials require wiping the state of Cloudiator, see [troubleshooting](../troubleshooting/README.md#reset-mowgli) ! ### OpenStack As first, we have to register a OpenStack cloud into the Mowgli framework. In order to do so, please make sure: * if you have one or more private networks, get the ID of the preferred network * volumes are not supported, flavors with root disks are required (>=70GB required for the scenarios described in the Mowgli paper ) * check if you are using either OpenStack with **Keystone** V2 or OpenStack V3 * get the **Keystone (identity)** endpoint URL of your OpenStack API, e.g. https://OPENSTACK_API:PORT/v3 * create the two security groups `mowgli-default` and `jclouds-mowgli`. The Mowgli framework will attach these security groups each spawned DBMS node (depending on the target cloud) * `mowgli-default` and `jclouds-mowgli` must allow all traffic within the private network. In addition the ports `22,1099,4369,33033` need to be open to the public. * See below an OpenStack example: ![security](../misc/security.png) * In case you want to use public IPs instead you need to allow *all* traffic or specify the required ports for each DBMS manually. Yet, we recommend to use only private IPs for realistic evaluation scenarios. * all parameters can be found in the **OpenStack RC File** that can be downloaded in the OpenStack dashboard under **API Access** #### OpenStack V2 ##### Web interface * Go to http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost * Enter the following parameters: ``` api: OPENSTACK apiVersion: V2 user: PROJECT_NAME:USER_NAME secret: xxxx apiEndpoint: https://OPENSTACK_API:PORT/v2.0 ``` * If you have more than one network you need to set the preferred network in the body: ``` [ { "configKey": "NETWORK", "configValue": "OPENSTACK_NETWORK_ID" } ] ``` * Otherwise pass an empty array in the body: ``` [] ``` * Click on *Try*. * The following message indicates the successful registration of the cloud: ``` { "id": 1, "name": "openstack-19-12-2018_09:46:36" } ``` #### OpenStack V3 ##### Web interface * Go to http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost * Enter the following parameters: ``` api: OPENSTACK apiVersion: V3 user: USER_DOMAIN:PROJECT_NAME/PROJECT_ID:USER_NAME secret: xxxx apiEndpoint: https://OPENSTACK_API:PORT/v3 ``` * USER_DOMAIN will be most likely `default` * If you have more than one network you need to set the preferred network in the body: ``` [ { "configKey": "NETWORK", "configValue": "OPENSTACK_NETWORK_ID" } ] ``` * Otherwise pass an empty array in the body: ``` [] ``` * Click on *Execute*. * The following message indicates the successful registration of the cloud: ``` { "id": 1, "name": "openstack-19-12-2018_09:46:36" } ``` ### Amazon AWS EC2 Currently Cloudiator does not support EBS volumes on EC2. This requires that the creation of a custom flavour with a predefined EBS volume following these steps: 1. Select the Ubuntu Server 16.04 as base image ![ec2 step1](../misc/step1.png) 2. Select a flavor (t2.medium recommended as this was used in the paper) ![ec2 step2](../misc/step2.png) 3. Use the default settings ![ec2 step3](../misc/step3.png) 4. Specify the disk size and type for the flavor and launch the instance ![ec2 step4](../misc/step4.png) 5. Create an image of the running instance which will be used as custom flavor ![ec2 step5](../misc/step5.png) 6. Apply valid name to the image that includes `ubuntu` and `16` so it will be discovered by Mowgli ![ec2 step6](../misc/step7b.png) 7. Create the custom image and keep the ami-ID which will be required later ![ec2 step7](../misc/step8.png) To allow the communication between all nodes in your private network extend the default security group to allow all traffic within the default security group. ![ec2 step8](../misc/step9.png) Register your cloud via the http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost with the settings: ``` api: EC2 apiVersion: DEFAULT user: user-id secret: api-key apiEndpoint: ``` If you have more than one network you need to set the preferred network in the body as `configValue`. In addition you need to set the image filter to only get the previously created image (otherwise Cloudiator will fetch all public images of EC2 which will take ages!) ``` [ { "configKey": "NETWORK", "configValue": "EC2_NETWORK_ID" }, { "configKey": "AMIQUERY", "configValue": "ami-093f8fa74f6c6b180" }, { "configKey": "AMICCQUERY", "configValue": "ami-093f8fa74f6c6b180" } ] ``` The following message indicates the successful registration of the cloud: ``` { "id": 1, "name": "ec2-19-12-2018_09:46:36" } ``` ------ You can verify if the registration of the cloud credentials was successful by checking the logs of the `mowgli_cloudiator-service_1` container via Portainer. In case you see don't see any error messages you are [now ready to run your first evaluations](../usage/README.md) :trophy: In case you see any error messages you need to wipe the Cloudiator database as described in the [troubleshooting](../troubleshooting/README.md) and rerun the cloud credentials registration with the correct credentials!