Data store connection configuration

A configuration file or application configuration must be provided in order for the DPS toolkit functions and operators to connect to external data stores.

A configuration consists of the data store type and store-specific fields (separated by a :).

The configuration precedence is:
  1. Value specified by dpsSetConfigFile(rstring)
    1. The value is first checked as a file (with relative paths relative to the toolkit/application root directory).
    2. If no file is found, the value is checked as a Streams application configuration.
  2. If no value was not specified, then the default value etc/no-sql-kv-store-servers.cfg is used.
The following are supported data store types:
  • redis

    A redis configuration works with a non-cluster redis instance in TLS or non-TLS mode.

    redis has the following store-specific fields (one server per line)
    
    RedisServerNameOrIPAddress:Port:RedisPassword:ConnectionTimeoutValue:UseTLS:RedisUsername
    
    Table 1. Redis Configuration

    Field

    Required

    Description

    Default

    RedisServerNameOrIPAddress

    X

    Server name or IP address

    Port

    X

    Server network port

    RedisPassword

    Database or user password

    ConnectionTimeoutValue

    Connection timeout (in seconds)

    3

    UseTLS

    1 = TLS; 0 = no TLS

    0

    RedisUsername

    Redis user

    The following is an example configuration of three redis servers with varying field values:
    
    Machine1:7001:MyRedisPassword:7:1:MyRedisUsername
    Machine2:7002:::0
    Machine3:7003:MyRedisPassword:6
    

    If you prefer to use the Unix domain socket instead of TCP, you can simply specify unixsocket instead of a server name. If you decide to use a unix domain socket, you must also ensure that your redis.conf file on the server side is configured properly for a unix domain socket pointing to /tmp/redis.sock file.

    NOTE: If you have a heavy workload of put/get requests, then using multiple Redis servers may scale to improve the overall throughput. If you use multiple Redis servers that are not configured in a redis-cluster mode, the DPS toolkit will automatically do the client-side partitioning to shard (i.e. spread) your data across those servers.

  • redis-cluster

    A redis-cluster configuration works with a redis cluster only in non-TLS mode. The other two redis configurations makes this one redundant though is kept for existing applications.

    redis-cluster has the following store-specific fields (one server per line)
    
    RedisServerNameOrIPAddress:port:RedisClusterPassword:RedisClusterUsername
    
    Table 2. Redis-cluster Configuration

    Field

    Required

    Description

    Default

    RedisServerNameOrIPAddress

    X

    Server name or IP address

    Port

    X

    Server network port

    RedisClusterPassword

    Database or user password

    RedisClusterUsername

    Redis user

    The following is an example configuration of four redis primary and replica nodes:
    
    Machine1:7001:MyRedisPassword:MyRedisUsername
    Machine2:7001:MyRedisPassword:MyRedisUsername
    Machine3:7001:MyRedisPassword:MyRedisUsername
    Machine4:7001:MyRedisPassword:MyRedisUsername
    
  • redis-cluster-plus-plus

    A redis-cluster-plus-plus configuration works with a redis cluster in TLS or non-TLS mode.

    redis-cluster-plus-plus has the following store-specific fields (one server per line)
    
    RedisServerNameOrIPAddress:port:RedisClusterPassword:ConnectionTimeoutValue:UseTLS:RedisClusterTlsCertificateFileName:RedisClusterTlsKeyFileName:RedisClusterCACertificateFileName:RedisClusterUsername
    
    Table 3. Redis-cluster-plus-plus Configuration

    Field

    Required

    Description

    Default

    RedisServerNameOrIPAddress

    X

    Server name or IP address

    Port

    X

    Server network port

    RedisClusterPassword

    Database or user password

    ConnectionTimeoutValue

    Connection timeout (in seconds)

    3

    UseTLS

    1 = TLS; 0 = no TLS

    0

    RedisClusterTlsCertificateFileName

    TLS certificate filename

    RedisClusterTlsKeyFileName

    TLS key filename

    RedisClusterCACertificateFileName

    TLS CA certificate filename

    RedisClusterUsername

    Redis user

    The following is an example of four redis primary and replica nodes:
    
    Machine1:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername
    Machine2:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername
    Machine3:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername
    Machine4:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername
    

Configuration file method

For configuration files, the data store type must be specified in the first uncommented line. Store-specific fields are provided on subsequent lines.

The following is an example config file for the redis type:


# This file is a simple configuration file for the Streams DPS toolkit.
# Any line starting with # is a comment line.
# Specify the name of the data store in the first non-comment regular line of this file
redis
#redis-cluster
#redis-cluster-plus-plus

# Specify the store-specific fields servers' names or IP addresses, port numbers, passwords and other fields as applicable.
#redis:
#  RedisServerNameOrIPAddress:Port:RedisPassword:ConnectionTimeoutValue:UseTLS:RedisUsername
Machine1:7001:MyRedisPassword:7:1:MyRedisUsername

#redis-cluster:
#  RedisServerNameOrIPAddress:port:RedisClusterPassword:RedisClusterUsername

#redis-cluster-plus-plus:
#  RedisServerNameOrIPAddress:port:RedisClusterPassword:ConnectionTimeoutValue:UseTLS:RedisClusterTlsCertificateFileName:RedisClusterTlsKeyFileName:RedisClusterCACertificateFileName:RedisClusterUsername

A full copy of the configuration file can be found in any of the DPS sample applications.

App config method

For application configurations, the data store type must be specified with property key dps.nosql.db and each store-specific config with property key dps.serverN.

The following is an example of how a DPS application configuration named dps-cfg-for-app1 would be created using streamtool:


streamtool mkappconfig --property dps.nosql.db=redis dps-cfg-for-app1   
streamtool chappconfig --property dps.server1=MyMachine1:6379:MyPassword dps-cfg-for-app1
streamtool chappconfig --property dps.server2=MyMachine2:6379:MyPassword dps-cfg-for-app1
streamtool chappconfig --property dps.server3=MyMachine3:6379:MyPassword dps-cfg-for-app1
streamtool chappconfig --property dps.server4=MyMachine4:6379:MyPassword dps-cfg-for-app1