Processing element startup problems

If the processing elements fail to start after you submit an application job, you might have to increase the value of the hc.pecStartTimeout property for the domain controller service.

You can encounter the following symptoms:

When the problem is the hc.pecStartTimeout property for the domain controller service, the processing element log file contains SIGTERM messages, as shown in the following example. The log file name is pec.pepe-id.stdouterr, where pe-id is the processing element ID.

============================  SIGNAL SIGTERM     ============================
/opt/ibm/InfoSphereStreamsZZZ/bin/../lib/libstreams-apputils.so.0(Distillery::BacktraceDumper::dump(int, bool, int)+0x46) [0x2b26d297098e]
/opt/ibm/InfoSphereStreamsZZZ/bin/../lib/libstreams-runtime.so.0(Streams::PECDaemon::sigTermHandler(int)+0x76) [0x2b26d17f86e0]
/opt/ibm/InfoSphereStreamsZZZ/bin/../lib/libstreams-runtime.so.0(Distillery::MemberFunction_1<Streams::PECDaemon, int>::call(int)+0x74) [0x2b26d17fae76]
/opt/ibm/InfoSphereStreamsZZZ/bin/../lib/libstreams-runtime.so.0(Distillery::Callback_1<int>::call(int)+0x3b) [0x2b26d17fadff]
/opt/ibm/InfoSphereStreamsZZZ/bin/../lib/libstreams-runtime.so.0(Distillery::Callback_1<int>::operator()(int)+0x26) [0x2b26d20f529c]
/opt/ibm/InfoSphereStreamsZZZ/bin/../lib/libstreams-runtime.so.0(Distillery::DistilleryApplication::errorSignalHandler(int)+0x177) [0x2b26d20e70d5]
... ...
... ...

When the problem involves the value of the DISPLAY environment variable and a config wrapper in the application, the processing element stays in the Starting state for awhile and then the processing element fails and the state of the processing element changes to Stopped. You can use the streamtool lspes command to view the status of the processing element.
To view the DISPLAY environment variable in the console log file, enter the following command:
```
streamtool viewlog --which console --pe pe-id --print
```
The Streams Console log for the processing element displays a message that is similar to the following example, which is taken from the WordCount sample application. To replicate the behavior of the WordCount sample application that you see in this example, ensure that the application uses the DISPLAY environment variable by adding a config wrapper to the sample.
```
$ (DISPLAY=myBadDisplayValue; 
streamtool submitjob -P file=/etc/hosts output/sample.WordCount.adl)
$
<wait for PE failure>
$ streamtool viewlog --which console --pe pe-id --print
  xterm Xt error: Can't open display: myBadDisplayValue
```

The following conditions can cause these problems:

When the problem is the hc.pecStartTimeout property for the domain controller service, the problem is typically the result of a slow network or an overloaded host. When the controller starts a processing element container, the value of the hc.pecStartTimeout property is equal to the maximum time that the controller waits for the processing element container to start. If the processing element container does not start within the allotted timeout period, the processing element fails to start.
When the problem involves the value of the DISPLAY environment variable and a config wrapper in the application, this problem occurs because the value of the DISPLAY environment variable is not valid.

To resolve these problems:

When the problem is the hc.pecStartTimeout property for the domain controller service, increase the value of the property. The default value is 30 seconds. If you increase the value of this property and the problem is not resolved, you can enable the host load protection or update the value of the host load threshold.
When the problem involves the value of the DISPLAY environment variable and a config wrapper in the application, see the description of the wrapper config.