Skip to content
GitLab

Scalability Evaluation · Changes

Page history
Update Scalability Evaluation authored by Daniel Seybold's avatar Daniel Seybold
Hide whitespace changes
Inline Side-by-side
Scalability-Evaluation.md
View page @ fccd63d5
Sensor Storage
Evaluating the Scalability (throughput,latency in correlation to cluster size) of DBMS cluster.
###### Web interface
#### Evaluation Scenario Workflow
1. Go to http://MOWGLI_HOST:8282/#/default/scenarioSensorStorageClusterPost
2. Click on *Try Out*
3. Provide these parameter about details of your evaluation run:
```
#the following parameters will be used to create the result folder structure
scenarioType: SENSORSTORAGE
cloud: <OPENSTACK/EC2>
dbmsType: <CASSANDRA/COUCHBASE>
dbmsConfiguration: short name for the applied config, e.g. 3nodes-weakConsistency
The evaluation scenario execution comprises the following steps:
#the follwoing parameters will be used to control the evaluation execution
runs: <1-n> the number of evaluation execution for the specified scenario
cleanUp: <true/false> specifies the deletion of each DBMS VM after an executed scenario (true is recommended, false only for debugging)
```
1. deploy *DBMS_X* in cluster size *C_Y* on cloud resources *CR_Z*
2. execute workload *W_A*
3. process performance and system metrics in relation to cluster size
For the actual scenario specification you find in the examples folder the evaluation scenarios presented in the paper. Just copy the JSON into the body and replace the the `TODO` values of `idCloud, idImage, idHardware, idLocation` (for databaseDataComponent and databaseSeedComponent) with the values of the previously fetched VM template.
#### Supported DBMS
Hit the button **Execute** and grab a :coffee: :tea:
- Cassandra
- CockroachDB (beta)
- Couchbase
- Elasticsearch
- MongoDB (beta)
- Riak
- Yugabyte (beta)
###### Shell
##### Important DBMS Parameters
- databaseSeedComponent instances -> 1 seed instance + databaseSeedComponent instances define the cluster size
- ReplicationFactor -> 1..clustersize-1
1. Use the following call, change the first 8 lines according to your cloud.
```
RESOURCETYPE=VM
IDCLOUD=1
IDIMAGE=97
IDHARDWARE=112
IDLOCATION=122
SCENARIO_TYPE=SENSORSTORAGE
CLOUD=OPENSTACK
DBMS_TYPE=CASSANDRA
DBMS_CONFIG_NAME=some_random_name
RUNS=1
CLEANUP=false
MOWGLI_IP=x.x.x.x
```
#### Supported Workloads
#### YCSB
- Sensor Storage (YCSB write only)
- YCSB (multi-phase: write + CRUD phase)
##### Important YCSB Parameters
- DBMS specific Read/Write consistency settings, please check the [YCSB DBMS bindings](https://github.com/brianfrankcooper/YCSB) for more details
curl -X POST "http://$MOWGLI_IP:8282/v1/scenario/sensorStorageCluster?scenarioType=$SCENARIO_TYPE&cloud=$CLOUD&dbmsType=$DBMS_TYPE&dbmsConfiguration=$DBMS_CONFIG_NAME&runs=$RUNS&cleanUp=$CLEANUP" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"workloadNetwork\": { \"type\": \"PRIVATE\" }, \"dbmsNetwork\": { \"type\": \"PRIVATE\" }, \"dbmsCluster\": { \"type\": \"CASSANDRA\", \"databaseDataComponent\": [ { \"instances\": 2, \"resource\": { \"resourceType\": \"$RESOURCETYPE\", \"idCloud\": $IDCLOUD, \"idImage\": $IDIMAGE, \"idHardware\": $IDHARDWARE, \"idLocation\": $IDLOCATION }, \"customConfiguration\": [ ], \"name\": \"DATA\", \"nodeConfiguration\": { \"dataMemory\": { \"envName\": \"DATAMEMORY\", \"envValue\": 0 }, \"indexMemory\": { \"envName\": \"INDEXMEMORY\", \"envValue\": 0 } } } ], \"databaseSeedComponent\": [ { \"instances\": 1, \"resource\": { \"resourceType\": \"$RESOURCETYPE\", \"idCloud\": $IDCLOUD, \"idImage\": $IDIMAGE, \"idHardware\": $IDHARDWARE, \"idLocation\": $IDLOCATION }, \"customConfiguration\": [ ], \"name\": \"SEED\", \"replicationFactor\": { \"envName\": \"REPLICATIONFACTOR\", \"envValue\": 3 }, \"nodeConfiguration\": { \"dataMemory\": { \"envName\": \"DATAMEMORY\", \"envValue\": 1500 }, \"indexMemory\": { \"envName\": \"INDEXMEMORY\", \"envValue\": 500 } } } ], \"databaseManagementComponent\": [ ] }, \"workload\": { \"dbEndpoints\": [ ], \"measurementConfig\": { \"interval\": 10, \"measurementType\": \"NONE\" }, \"workloadConfig\": { \"workloadType\": \"LOAD\", \"workloadClass\": \"com.yahoo.ycsb.workloads.CoreWorkload\", \"maxExecutionTime\": 1800, \"threadCount\": 16, \"recordCount\": 4000000, \"insertStart\": 0, \"insertCount\": 0, \"operations\": 1000, \"targetOps\": 0, \"fieldCount\": 10, \"fieldLength\": 500, \"readAllFileds\": true, \"readModifyWriteProportion\": 0, \"requestdistribution\": \"UNIFORM\", \"scanLengthDistribution\": \"UNIFORM\", \"insertOrder\": \"ORDERED\", \"readProportion\": 0.0, \"updateProportion\": 0.0, \"insertProportion\": 1.0, \"scanProportion\": 0, \"maxScanLength\": 1000, \"coreWorkloadInsertionRetryLimit\": 3, \"coreWorkloadInsertionRetryInterval\": 3 }, \"databaseConfig\": { \"databaseBinding\": \"CASSANDRA2\", \"endpointParameterName\": \"hosts\", \"tableParameterName\": \"cassandra.keyspace\", \"tableName\": \"ycsb\", \"configPorperties\": [ { \"name\": \"cassandra.writeconsistencylevel\", \"value\": \"ONE\" } ] } }}"
2. The output signalizes the successful run (after several minutes):
```
{
"code": 4,
"type": "ok",
"message": "Sensor Storage Cluster evaluation finished!"
}
```
#### TPC-C
- TPC-C (by Cockroach loadgen)
### After evaluation start
Now the execution of the evaluation is started and you will see the VMs coming up in your cloud dashboard. In addition you can follow the logs of the evaluation via portainer: login via http://MOWGLI_HOST:9001/#/auth with the default credentials user: `admin` and password `mowgli19`
Check the logs of the evaluation-orchestrator container and follow the progress of the evaluation.
#### Supported Clouds
You can also check the resource consumption via the Chronograf dashboard (if port 8888 is exposed):
- OpenStack V2
- OpenStack V3
- Amazon EC2
http://MOWGLI_HOST:8888
After a successful execution you will find the results under:
```
/opt/evaluation-results/sensorstorage/CLOUD/DBMS/CONFIGURATION
```
#### Examples
Feel free to run multiple evaluation scenarios (in case you have enough EC2 credits :smirk: ) and play with the following template options:
```
instances: X (only modify the number of databaseDataComponent!)
replicationfactor: 1-numberOfNodes
maxExecutionTime: 60-3600 (unit seconds)
recordCount: 10000 - n (make sure your VM have enough disk space, 4000000 require around 60GB of disk per Cassandra node)
fieldCount: 10 (each field is 500 bytes)TODO: explain the mowgli-default security group in openstack which might need some manual adjustments
```
YCSB Multi Phase
Plotting
In order to generate plots to visualize the results you can use the processing call http://MOWGLI_IP:8282/#/default/processingBoxplotPost
You need to specify a plot name and a map of absolute evaluation result folders and labels (not more than 5 characters) .
Example:
```
plotName: icpe
[
{
"resultFolder": "/opt/evaluation-results/sensorstorage/ec2/cassandra/myConfig1",
"label": "conf1"
},
{
"resultFolder": "/opt/evaluation-results/sensorstorage/ec2/cassandra/myConfig2",
"label": "conf2"
},
{
"resultFolder": "/opt/evaluation-results/sensorstorage/ec2/cassandra/myConfig3",
"label": "conf3"
}
]
```
You will find the plot under `/opt/evaluation-results/sensorstorage/plotName.pdf`
For getting started please have a look at the scalability [example templates](https://omi-gitlab.e-technik.uni-ulm.de/mowgli/getting-started/tree/master/examples/scalability)
\ No newline at end of file