Example: Using resource usage thresholds and PE restart recommendations to balance workloads

Scenario: You see that the workload is not balanced across resources. You want to move PEs from the resources that are overloaded to a resource that is underloaded.

About this task

You can change the resource usage thresholds and restart PEs with streamtool commands.

Procedure

View the current metrics for each resource in your system by running the streamtool capturestate command. Look for the values of these metrics under the <host> element:

cpuUtilization
memoryUtilization
networkUtilization

Example command:

streamtool capturestate --select hosts=metrics

On one of the resources, you see that the CPU usage and memory usage are high:

…
    <metric name="cpuUtilization" lastChangeObserved="1504302827" userDefined="false">
      <metricValue xsi:type="streams:longType" value="90"/>
    </metric>
…
    <metric name="memoryUtilization" lastChangeObserved="1504302827" userDefined="false">
      <metricValue xsi:type="streams:longType" value="90"/>
    </metric>
   …
    <metric name="networkUtilization" lastChangeObserved="1504302827" userDefined="false">
      <metricValue xsi:type="streams:longType" value="15"/>
    </metric>
…

Modify the upper thresholds for CPU usage and memory usage:

streamtool setproperty resourceCpuUsageUpperThreshold=75 resourceMemoryUsageUpperThreshold=50

Run the streamtool lsrestartrecs command to view PE restart recommendations for the instance. Example output:
```
Threshold: INSTANCE
  Restart Priority JobName                       PE Current Resource
  none             application::TradesAppCloud_0 0  streamsqse.localdomain
  preferred        application::TradesAppCloud_0 1  streamsqse.localdomain 
  high             application::TradesAppCloud_0 2  streamsqse.localdomain 
```
Processing element ID 1 has a preferred recommendation, which means that it is the best candidate to be restarted. Processing element ID 2 has a high recommendation, which means the current resource running the PE is overloaded and there is at least one candidate resource that is underloaded.
Stop the processing elements ID 1 and ID 2:
```
streamtool stoppe 1 2
```
Restart the processing elements ID 1 and ID 2:
```
streamtool restartpe 1 2
```
After a representative amount of the workload has been processed, repeat step 1 to confirm that moving the PEs from overloaded resources to underloaded resources provides the load balancing that you want. You can then optionally modify the thresholds again and view and run the PE restart recommendations.