SPL File CrossDataCenterFailover.spl

This Streams SPL composite operator contains the logic to do cross data center failover for a given Streams application.

It provides a generic framework for detecting planned or unplanned data center outages and allowing the surviving data center to continue the operational responsibilities of the data center that went down a few moments earlier. To achieve that goal, it performs the following major tasks:

1) It exchanges periodic heartbeat messages with the configured remote data center.

2) During the hearbeat exchange process, it looks out for the prolonged heartbeat misses from the remote data center so as to plan for the operational failover (or switchover) to the surviving data center.

3) It notifies the external application logic whenever a remote data center's status transition occurs from being DOWN to UP or UP to DOWN.

4) It also (optionally) exchanges the application-specific in-memory data snapshot with the remote data center. This snapshot will ensure that any in-flight in-memory state data kept at the failed data center to get processed by the other surviving data center. In essence, it does the cross data center replication of the in-memory state maintained by the external applications. This data replication is needed for the operational continuity when one of the data centers goes down in a planned or unplanned manner. This composite provides a data agnostic mechanism for the external applications to exchange (replicate) their proprietary data across the data centers.

5) Upon detecting a remote data center failure or inactivity, it signals the external application logic to take over the remote data center's replicated data snapshot and apply the necessary processing/application logic to that data.

6) Details discussed above are for two data centers working in active-active mode i.e. two copies of the same application actively processing data in both DCs. This toolkit can also be used for active-passive (or active-standby) mode in which one DC is actively processing data and the other DC is simply running some code similar to this toolkit's test driver code as a standalone application with the second copy of the real application not at all running. In this case, this stub application running in standalone mode will ignore any data snapshot signals. Whenever it gets a notification about the remote active DC failure, this stub application will simply launch a shell script to do "streamtool submitjob" of the actual application in distributed mode. At that point, the standalone stub application code can be stopped either within that shell script or by some other means. (Please refer to a sample project in this toolkit directory named CrossDataCenterFailoverPassiveSample.)

Any application that wants to make use of this cross data center failover technique has to do the following:

a) It must use the use clauses to include the toolkit namespaces and add the toolkit to the Streams compiler via -t <path-to-toolkit>.

b) It must hook with this composite in this project via the appropriate input and output streams to enable the failover option between the two data centers. A given external application must also ensure that the composite parameters specified below are configured correctly in the Streams app config facility.

c) Also note that there are four C++ native functions named serializeTuple to serialize a tuple into binary payload, deserializeTuple to deserialize a given blob into a tuple, serializeDataItem to serialize any data item made of any SPL type into a blob and deserializeDataItem to deserialize a given blob into its original data item made of any SPL type. Those native functions also need to be used within the external applications wanting to make use of this cross data center failover technique. Look inside the impl/include/Functions.h and in the com.teracloud.streams.crossdc.failover/native.function/function.xml files for the C++ code and the XML function model respectively.

d) There is also a custom Java operator (HttpBlobSender) in this project which is mainly used within this composite. It will not be directly used by the other external applications making use of this composite. But, it is necessary to understand how that operator works.

Example
The following is a code block that demonstrates how to invoke the CrossDataCenterFailover composite operator within an Streams application:

Content

Operators

Composites

composite CrossDataCenterFailover(output DataSnapshotSignal, RemoteDataCenterStatus, ProcessDataFromRemoteDC; input SerializedDataSnapshotMessage, SpecialMessage)

SPL composite operator image not displayed. Problem loading file: ../../image/tk$com.teracloud.streams.crossdcfailover/op$com.teracloud.streams.crossdc.failover$CrossDataCenterFailover.svg

Parameters

  • localDataCenterName
  • crossDCOperationMode
  • crossDCHttpPort
  • remoteDataCenterApplicationMachineNames
  • crossDCInitDelay
  • heartbeatExchangeInterval
  • consecutiveHeartbeatMissesAllowed
  • dataSnapshotExchangeInterval
  • dataSnapshotStorageDirectory
  • dataSnapshotJdbcUrl
  • dataSnapshotJdbcUser
  • dataSnapshotJdbcPassword
  • dataSnapshotJdbcDriverLib
  • dataSnapshotJdbcClassName
  • dataSnapshotTableName
  • dataSnapshotPrimaryKeyColumnName
  • sendToAllRemoteMachines
  • alwaysLogHttpErrors
  • httpConnectionTimeout
  • httpReadTimeout
  • numberOfHttpSenderReceiverPairs
  • remoteDataCenterUrlPathForHeartbeatReceiver
  • remoteDataCenterUrlPath
  • retainOlderDataSnapshotsAtStartup
  • sendDataSnapshotsToOriginDCAtStartup
  • configFileName
  • localDataCenterNameKey
  • crossDCOperationModeKey
  • crossDCHttpPortKey
  • remoteDataCenterApplicationMachineNamesKey
  • dataSnapshotStorageDirectoryKey
  • dataSnapshotJdbcUrlKey
  • dataSnapshotJdbcUserKey
  • dataSnapshotJdbcPasswordKey
  • dataSnapshotJdbcDriverLibKey
  • dataSnapshotJdbcClassNameKey
  • dataSnapshotTableNameKey
  • dataSnapshotPrimaryKeyColumnNameKey
  • crossDCInitDelayKey
  • heartbeatExchangeIntervalKey
  • consecutiveHeartbeatMissesAllowedKey
  • dataSnapshotExchangeIntervalKey
  • sendToAllRemoteMachinesKey
  • alwaysLogHttpErrorsKey
  • httpConnectionTimeoutKey
  • httpReadTimeoutKey
  • retainOlderDataSnapshotsAtStartupKey
  • sendDataSnapshotsToOriginDCAtStartupKey

Input Ports

  • SerializedDataSnapshotMessage
  • SpecialMessage

Output Ports

  • DataSnapshotSignal
  • RemoteDataCenterStatus
  • ProcessDataFromRemoteDC