README.md 11.8 KB
Newer Older
1 2 3 4 5 6
## 1. Setup Mowgli

The Mowgli components are encapsulated as Docker containers and bundle via Docker-Compose.

The docker-compose contains all required components that are highlighted in blue.

7
![mowgli-setup](../misc/mowgli_architecture_spec_2.png)
8

Daniel Seybold's avatar
Daniel Seybold committed
9
While the Mowgli framework can be run locally or on a VM, the Workload-API needs to run on a separate VM within the selected cloud (to use the private network and to ensure isolated workload creation). The default setup includes the Workload-API and assumes that the Mowgli framework is deployed in the same cloud and network that the evaluations are targeting.  Additional Workload-APIs can added if required by following the [Workload-API installation guide](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Deploy-Workload-API) . 
10

Daniel Seybold's avatar
Daniel Seybold committed
11
In the following, the required steps to setup, configure and use the Mowgli Framework are described. 
12

Daniel Seybold's avatar
Daniel Seybold committed
13
**Requirement:** An account for an OpenStack-based cloud or the AWS EC2 cloud.
14

Daniel Seybold's avatar
Daniel Seybold committed
15
This guide consists of the following steps:
16

Daniel Seybold's avatar
Daniel Seybold committed
17 18
1. Install Mowgli  

19 20
2. Prepare and register the target cloud in Mowgli  
   
Daniel Seybold's avatar
Daniel Seybold committed
21
   2.1. OpenStack  
22 23
2.2. Amazon AWS EC2  
   
Daniel Seybold's avatar
Daniel Seybold committed
24 25 26 27 28 29
   




#### 1. Install and start Mowgli Framework
30 31 32 33 34 35 36 37 38

As the Mowgli Framework comprises multiple dockerized components, Mowgli is provided as docker-compose bundle. 
Though, the Mowgli framework can be deployed locally if the host is publicly accessible (unique IP and required ports open) and the Docker and Docker-Compose are installed

Yet, we recommend to create a dedicated VM. 
The IP of the host of the Mowgli framework is denoted in the following as *MOWGLI_IP*.

| property                 | required                                                     |
| ------------------------ | ------------------------------------------------------------ |
Daniel Seybold's avatar
Daniel Seybold committed
39
| image / operating system | Ubuntu 18.04 (recommended), any Linux-based OS with docker and docker-compose installed will work |
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57
| CPU cores                | 4                                                            |
| memory                   | 4GB                                                          |
| disk                     | 20GB                                                         |
| software                 | Docker, Docker-Compose                                       |
| open ports               | 3389 - RMI for the DBMS deployment via Cloudiator            |
|                          | 4001 - ETCD to store Cloudiator deployment metadata          |
|                          | 7001 - ETCD to store Cloudiator deployment metadata          |
|                          | 8080 - the php-admin interface of the cloudiator dbms        |
|                          | 8181 - the endpoint of the Workload-API                      |
|                          | 8086 - the REST interface of InfluxDB to store the monitoring data |
|                          | 8282 - the basic web interface of Mowgli framework           |
|                          | 8888 - Chronograf, a web interface for the InfluxDB          |
|                          | 9000 - the REST interface of Cloudiator                      |
|                          | 9001 - Portainer: a simple management UI for the docker containers |
|                          | 9091 - the basic web interface of dbms-catalogue             |

Execute the following steps to deploy the Mowgli framework:

58
**Task 1**: install [docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) on your host system   
59 60 61
**Task 2:** you have to change to root user: ```sudo -s```  
**Task 3:** create the folder to store the evaluation results and plots: ```sudo mkdir -p /opt/evaluation-results/```  
**Task 4:** create the folder for the Cloudiator data: ```sudo mkdir -p /var/lib/cloudiator/cloudiator-store/```  
62
**Task 5:** checkout the repository which includes the docker-compose.yml: ```git clone -b 0.2  https://omi-gitlab.e-technik.uni-ulm.de/mowgli/docker.git mowgli```  
63
**Task 6:** create the .env file based on the env-template: ```cd mowgli &&  cp env-template .env```  
Daniel Seybold's avatar
Daniel Seybold committed
64
**Task 7:** add the **publicly accessible IP** of the host to the .env  
65
**Task 8 (optional):** In case the host is running in an enterprise network and requires certificates please check the [wiki](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/wikis/Add-Certificates)  
66 67 68 69 70
**Task 9:** start the Mowgli services via: ```sudo docker-compose up -d```  
**Task 10:** open your browser and check if the Mowgli framework started successfully at: `http://MOWGLI_HOST:8282`   or `http://MOWGLI_HOST:8282/explorer`

If you can access the web interface you are nearly there :smiley:

Daniel Seybold's avatar
Daniel Seybold committed
71
If not check the [troubleshooting](../troubleshooting/README.md) section :scream:
72 73 74 75 76 77

In addition Mowgli comes with the [Portainer](https://www.portainer.io/) tool to provide a graphical UI for the Docker containers. 

Portainer is accessible via http://MOWGLI_HOST:9001 with the default user *admin* and password *mowgli19*


Daniel Seybold's avatar
Daniel Seybold committed
78
## 2. Prepare and register the target Cloud in Mowgli 
79 80 81 82 83 84 85 86 87 88

In the following, we will prepare Mowgli to use your Cloud. For this, we provide
two options: 

1. Using the default web interface at ```http://MOWGLI_HOST_IP:8282/``` 
2. (recommended) Using the more responsive web interface at `http://MOWGLI_HOST:8282/explorer`

Mowgli enables the portable execution of evaluation scenarios over arbitrary OpenStack clouds and AWS EC2 by abstracting most of the technical details. 
Yet, there a few **requirements** due to usage of [Cloudiator](http://cloudiator.org/) as deployment engine:

Daniel Seybold's avatar
Daniel Seybold committed
89 90
- the cloud needs to offer an **Ubuntu 16 (recommended, Ubuntu 18 should also work)** image that has `wget` installed (which is by default) 
- the image name has to contain `ubuntu` and `16 ` (or `18`) to be selected for the deployment of the DBMS
91 92 93 94 95 96 97 98 99
- the default user of the image is `ubuntu`
- if your cloud does not provide an Ubuntu 16 image you can download one [here](https://cloud-images.ubuntu.com/xenial/) and upload it to your cloud
- Cloudiator can handle the security group management but it is required to allow all traffic within the private network to avoid connection issues. The required steps are explained in the following for OpenStack and EC2  


In the following, the steps to register a cloud within Mowgli are explained. 

**Note**: While your actual cloud credentials are required, they will never be exposed 
via the API (which currently also means invalid credentials require wiping the 
Daniel Seybold's avatar
Daniel Seybold committed
100
state of Cloudiator, see [troubleshooting](../troubleshooting/README.md#reset-mowgli)  !
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209



### OpenStack

As first, we have to register a OpenStack cloud into the Mowgli framework. In order to do so, please make sure:

* if you have one or more private networks, get the ID of the preferred network 

* volumes are not supported, flavors with root disks  are required (>=70GB required for the scenarios described in the Mowgli paper )

* check if you are using either OpenStack with **Keystone** V2 or OpenStack V3

* get the **Keystone (identity)** endpoint URL of your OpenStack API, e.g. https://OPENSTACK_API:PORT/v3

* create the two security groups `mowgli-default` and `jclouds-mowgli`. The Mowgli framework will attach these security groups each spawned  DBMS node (depending on the target cloud)

* `mowgli-default` and `jclouds-mowgli`  must  allow all traffic within the private network. In addition the ports `22,1099,4369,33033` need to be open to the public. 

* See below an OpenStack example: 

  ![security](../misc/security.png) 

* In case you want to use public IPs instead you need to allow *all* traffic or specify the required ports for each DBMS manually. Yet, we recommend to use only private IPs for realistic evaluation scenarios.

* all parameters can be found in the **OpenStack RC File** that can be downloaded in the OpenStack dashboard under **API Access** 

#### OpenStack V2

##### Web interface

* Go to http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost
* Enter the following parameters:
     ```
     api: OPENSTACK
     apiVersion: V2
     user: PROJECT_NAME:USER_NAME
     secret: xxxx
     apiEndpoint: https://OPENSTACK_API:PORT/v2.0
     ```
* If you have more than one network you need to set the preferred network in the body:
     ```
     [
       {
         "configKey": "NETWORK",
         "configValue": "OPENSTACK_NETWORK_ID"
       }
     ]
     ```
* Otherwise pass an empty array in the body:
     ```
     []
     ```
* Click on *Try*.
* The following message indicates the successful registration of the cloud:
     ```
     {
       "id": 1,
       "name": "openstack-19-12-2018_09:46:36"
     }
     ```

#### OpenStack V3


##### Web interface

* Go to http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost

* Enter the following parameters:
     ```
     api: OPENSTACK
     apiVersion: V3
     user: USER_DOMAIN:PROJECT_NAME/PROJECT_ID:USER_NAME
     secret: xxxx
     apiEndpoint: https://OPENSTACK_API:PORT/v3
     ```

* USER_DOMAIN will be most likely `default`

* If you have more than one network you need to set the preferred network in the body:

     ```
     [
       {
         "configKey": "NETWORK",
         "configValue": "OPENSTACK_NETWORK_ID"
       }
     ]
     ```

* Otherwise pass an empty array in the body:
     ```
     []
     ```

* Click on *Execute*.

* The following message indicates the successful registration of the cloud:
     ```
     {
       "id": 1,
       "name": "openstack-19-12-2018_09:46:36"
     }
     ```


### Amazon AWS EC2

Daniel Seybold's avatar
Daniel Seybold committed
210
Currently Cloudiator does not support EBS volumes on EC2. This requires that the creation of a custom flavour with a predefined EBS volume following these steps:
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286

1. Select the Ubuntu Server 16.04 as base image

   ![ec2 step1](../misc/step1.png)

2. Select a flavor (t2.medium recommended as this was used in the paper)

   ![ec2 step2](../misc/step2.png)

3. Use the default settings 

   ![ec2 step3](../misc/step3.png)

4. Specify the disk size and type for the flavor and launch the instance

![ec2 step4](../misc/step4.png)

5. Create an image of the running instance which will be used as custom flavor

   ![ec2 step5](../misc/step5.png)



6. Apply valid name to the image that includes `ubuntu` and `16` so it will be discovered by Mowgli 

   ![ec2  step6](../misc/step7b.png)

7. Create the custom image and keep the ami-ID which will be required later

   ![ec2 step7](../misc/step8.png)



To allow the communication between all nodes in your private network extend the default security group to allow all traffic within the default security group. 

![ec2 step8](../misc/step9.png)

Register your cloud via the http://MOWGLI_HOST_IP:8282/explorer/#!/management/templateCloudPost

with the settings:

```
api: EC2
apiVersion: DEFAULT
user: user-id
secret: api-key
apiEndpoint: 
```

If you have more than one network you need to set the preferred network in the body as `configValue`. In addition you need to set the  image filter to only get the previously created image (otherwise Cloudiator will fetch all public images of EC2 which will take ages!)

```
[
  {
    "configKey": "NETWORK",
    "configValue": "EC2_NETWORK_ID"
  },
  {
    "configKey": "AMIQUERY",
    "configValue": "ami-093f8fa74f6c6b180"
  },
  {
	"configKey": "AMICCQUERY",
    "configValue": "ami-093f8fa74f6c6b180"
  }

]
```

The following message indicates the successful registration of the cloud:

```
{
  "id": 1,
  "name": "ec2-19-12-2018_09:46:36"
}
Daniel Seybold's avatar
Daniel Seybold committed
287 288 289 290
```

------

Daniel Seybold's avatar
Daniel Seybold committed
291 292 293 294
You can verify if the registration of the cloud credentials was successful by checking the logs of the `mowgli_cloudiator-service_1` container via Portainer. 

In case you see don't see any error messages you are [now  ready to run your first evaluations](../usage/README.md) :trophy: 

Daniel Seybold's avatar
Daniel Seybold committed
295
In case you see any error messages you need to wipe the Cloudiator database as described in the [troubleshooting](../troubleshooting/README.md) and rerun the cloud credentials registration with the correct credentials!