Configuring a checkpoint data store for Teracloud® Streams domains and instances

This value specifies that Teracloud® Streams saves checkpoint data in a file system directory.

For fileSystem, the domain.checkpointRepositoryConfiguration or instance.checkpointRepositoryConfiguration properties specify an absolute path to an existing directory, which is the root checkpointing directory. If the instance needs to run jobs with restartable and relocatable operators, then place the checkpointing directory in a shared file system.

The value of checkpointRepositoryConfiguration is in a JSON format.

instance.checkpointRepositoryConfiguration = " { \"Dir\" : path } " 

path is the absolute path to an existing directory.

The following example sets the checkpoint store in the user's home directory.

instance.checkpointRepositoryConfiguration=" { \"Dir\" : \"/home/user/ckpt\" } "

You can set domain and instance properties by using the streamtool setdomainproperty and streamtool setproperty commands. Use the streamtool getdomainproperty and streamtool getproperty commands to display property values.

This value specifies that Teracloud® Streams uses a Redis checkpoint data store.

Redis is an open source key-value data store. To use this store for checkpointing, you need to set up your Redis servers and provide their location to Teracloud® Streams with host names and ports in the checkpointRepositoryConfiguration string. Multiple Redis servers can be configured and checkpoint data is distributed among the servers through sharding and replication.

The value of checkpointRepositoryConfiguration is in a JSON format.

{
    "replicas" : N, 
    "shards" : K,
    "replicaGroups" : [replicaGroup1, replicaGroup2, ..., replicaGroupN]
}

{
    "servers" : [HostAndPortOfServer1, HostAndPortOfServer2, ..., HostAndPortOfServerK], 
    "description" : StringOfOptionalDescription
}

The value of "servers" is a JSON array that contains K "host:port" for the K Redis servers in the replica group. host can be host name or IP address and port is the port number that is used by the Redis server. The "description" is optional.

instance.checkpointRepositoryConfiguration=" {
\"replicas\" : 3,
\"shards\" : 8,
\"replicaGroups\" : [
    { \"servers\" : [\"host0:port0\", \"host1:port1\", \"host2:port2\", \"host3:port3\",
      \"host4:port4\", \"host5:port5\",\"host6:port6\", \"host7:port7\"],
      \"description\" : \"rack1\" }, 
    { \"servers\" : [\"host8:port8\", \"host9:port9\", \"host10:port10\", \"host11:port11\",
      \"host12:port12\",\"host13:port13\", \"host14:port14\", \"host15:port15\"],
      \"description\" : \"rack2\" }, 
    { \"servers\" : [\"host16:port16\", \"host17:port17\", \"host18:port18\",
      \"host19:port19\", \"host20:port20\",\"host21:port21\", \"host22:port22\",
      \"host23:port23\"],  \"description\" : \"rack3\" }
]
}"

• redis for a secure platform configuration

  You can configure Teracloud® Streams to use Redis servers which have authentication enabled as a checkpoint data store. You can specify a Redis password as part of the domain or instance checkpointRepositoryConfiguration property, store this password, and pass it to SPL runtime in a secure way:

  1. Use the streamtool setproperty command to set the checkpointRepository string to Redis:

     streamtool setproperty -d $STREAMS_DOMAIN_ID -i $STREAMS_INSTANCE_ID checkpointRepository=redis

  2. Use the streamtool setrestrictedconfig command to store Redis passwords. If there are six Redis servers, for example, and every two of them share the same password, you can choose which property name you want to use to store the passwords, such as:

     streamtool setrestrictedconfig -d $STREAMS_DOMAIN_ID -i $STREAMS_INSTANCE_ID redisPassword1=passwdOne
     
     streamtool setrestrictedconfig -d $STREAMS_DOMAIN_ID -i $STREAMS_INSTANCE_ID redisPassword2=passwdTwo
     
     streamtool setrestrictedconfig -d $STREAMS_DOMAIN_ID -i $STREAMS_INSTANCE_ID redisPassword3=passwdThree
     

  3. Use the streamtool setproperty command to specify the normal checkpointRepositoryConfiguration property with the host, port, and password of the Redis servers:

     streamtool setproperty -d $STREAMS_DOMAIN_ID -i $STREAMS_INSTANCE_ID checkpointRepositoryConfiguration ="  { 
     \"shards\":2,
     \"replicas\":3,\
     \"replicaGroups\":[\
       {\"servers\":[\"host1:6379:redisPassword1\",\"host2:6379:redisPassword1\", \"host3:6379:redisPassword2\"]},\
       {\"servers\":[\"host4:6379:redisPassword2\",\"host5:6379:redisPassword3\", \"host6:6379:redisPassword3\"]} \
       ] \
     }  " 
     

This value specifies that Teracloud® Streams uses an IBM Cloud Object Storage checkpoint data store.

Cloud Object Storage is a highly scalable cloud storage service, designed for high durability, resiliency and security. To use this data store for checkpointing, you must set up your Cloud Object Storage service in IBM Cloud

After you create the service, you must provide service credentials to Teracloud® Streams. These service credentials include endpoint, bucket name, and configuration entry name that contains service keys, as part of the string for checkpointRepositoryConfiguration in JSON format. You can find the service credentials on the Cloud Object Storage service dashboard in IBM Cloud.

{
    "endpoint" : url, 
    "bucket" : name,
    "auth" : name
}

The value of "endpoint" is a string that specifies the URL of a private or public Cloud Object Storage S3 endpoint. The value of "bucket" is a string that specifies the name of the container to store checkpoint objects. The bucket parameter is optional. If not specified, streams-checkpoint is used as bucket name. If the bucket does not exist, a new bucket named streams-checkpoint is created. The value of "auth" is a string that specifies the configuration entry name that contains all key-related information. Objects in Cloud Object Storage are encrypted at rest. This technology individually encrypts each object using per-object generated keys.

{
    "endpoint" : "s3-api.us-geo.objectstorage.service.networklayer.com", 
    "bucket" : "streams-checkpoint",
    "auth" : "apiKey"
}