Processing element connectivity problems

When streaming data is not transmitted to or received by a processing element (PE), you can troubleshoot the problem by verifying the status of the PEs and resources.

If PEs are failing, you might notice that tuples are not flowing in your application graph or that your application is not generating the expected output. If you are monitoring the Teracloud® Streams instance by using the Streams Console or streamtool commands, there might be a problem if you see that some PEs are in a Stopped, Stopping, or Unknown state.

Another symptom of this type of problem is that the PE log contains connection failure messages that are not followed by connection success messages. For example: 
01 Sep 2010 12:06:40.113 [29363] ERROR tcp M[TCPSender.cpp:connectToServer:384]  - 
Connection attempt failed for 'my_port' retrying (1)

01 Sep 2010 12:26:51.637 [12631] ERROR rum_sender M[RUMSender.cpp:onConnEventT:150]  - 
Connection attempt failed for 'my_port' retrying
The following example shows the connection failure messages when they are followed by the connection success messages:
01 Sep 2010 12:06:40.113 [29363] ERROR tcp M[TCPSender.cpp:connectToServer:384]  - 
Connection attempt failed for 'my_port' retrying (1)
01 Sep 2010 12:06:54.974 [29363] ERROR tcp M[TCPSender.cpp:connectToServer:280]  - 
Connection successfully established for 'my_port' (139)

01 Sep 2010 12:26:51.637 [12631] ERROR rum_sender M[RUMSender.cpp:onConnEventT:150]  - 
Connection attempt failed for 'my_port' retrying
01 Sep 2010 12:26:58.757 [12631] ERROR rum_sender M[RUMSender.cpp:onConnEventT:132]  - 
Connection successfully established for 'my_port'
A network transport transmits the streaming data between the PEs. Problems occur when a PE that sends data cannot connect to a PE that receives data. These problems block the transmission of the streaming data. The most common causes of PE connectivity problems are:
  • The PE that is receiving the data fails, and it cannot be restarted.
  • The PE that is receiving the data does not respond to the PE that is sending the data.
  • The resource of the PE that is receiving the data fails, and the resource cannot respond to the PE that is sending the data.
  • Other connectivity issues.
To resolve the problem:
  1. Verify that the resources meet the configuration and network connectivity requirements. This step is particularly important if the connections never worked.
  2. Check the states of the PEs and the resources that are used in the connections. For example:
    • View status and health information for the PEs through the Streams Console or the streamtool lspes command.
    • View status information for the instances and resources through the Streams Console or the streamtool getinstancestate command.
  3. Identify which PE's are failing and analyze the PE logs to find the cause of the problem.
    • To identify which PEs are failing, use the streamtool lspes command.
    • To retrieve the PE logs, use the streamtool getlog command.
    Multiple PE failures on a single resource can indicate a resource connectivity problem.

If none of these steps resolve the problem, contact Teracloud® Suppport.