Data store connection configuration
A configuration file or application configuration must be provided in order for the DPS toolkit functions and operators to connect to external data stores.
A configuration consists of the data store type and store-specific fields (separated by a :).
- Value specified by dpsSetConfigFile(rstring)
- The value is first checked as a file (with relative paths relative to the toolkit/application root directory).
- If no file is found, the value is checked as a Streams application configuration.
- If no value was not specified, then the default value etc/no-sql-kv-store-servers.cfg is used.
- redis
A redis configuration works with a non-cluster redis instance in TLS or non-TLS mode.
redis has the following store-specific fields (one server per line)RedisServerNameOrIPAddress:Port:RedisPassword:ConnectionTimeoutValue:UseTLS:RedisUsername
Table 1. Redis Configuration Field
Required
Description
Default
RedisServerNameOrIPAddress
X
Server name or IP address
Port
X
Server network port
RedisPassword
Database or user password
ConnectionTimeoutValue
Connection timeout (in seconds)
3
UseTLS
1 = TLS; 0 = no TLS
0
RedisUsername
Redis user
The following is an example configuration of three redis servers with varying field values:Machine1:7001:MyRedisPassword:7:1:MyRedisUsername Machine2:7002:::0 Machine3:7003:MyRedisPassword:6
If you prefer to use the Unix domain socket instead of TCP, you can simply specify unixsocket instead of a server name. If you decide to use a unix domain socket, you must also ensure that your redis.conf file on the server side is configured properly for a unix domain socket pointing to /tmp/redis.sock file.
NOTE: If you have a heavy workload of put/get requests, then using multiple Redis servers may scale to improve the overall throughput. If you use multiple Redis servers that are not configured in a redis-cluster mode, the DPS toolkit will automatically do the client-side partitioning to shard (i.e. spread) your data across those servers.
-
redis-cluster
A redis-cluster configuration works with a redis cluster only in non-TLS mode. The other two redis configurations makes this one redundant though is kept for existing applications.
redis-cluster has the following store-specific fields (one server per line)RedisServerNameOrIPAddress:port:RedisClusterPassword:RedisClusterUsername
Table 2. Redis-cluster Configuration Field
Required
Description
Default
RedisServerNameOrIPAddress
X
Server name or IP address
Port
X
Server network port
RedisClusterPassword
Database or user password
RedisClusterUsername
Redis user
The following is an example configuration of four redis primary and replica nodes:Machine1:7001:MyRedisPassword:MyRedisUsername Machine2:7001:MyRedisPassword:MyRedisUsername Machine3:7001:MyRedisPassword:MyRedisUsername Machine4:7001:MyRedisPassword:MyRedisUsername
-
redis-cluster-plus-plus
A redis-cluster-plus-plus configuration works with a redis cluster in TLS or non-TLS mode.
redis-cluster-plus-plus has the following store-specific fields (one server per line)RedisServerNameOrIPAddress:port:RedisClusterPassword:ConnectionTimeoutValue:UseTLS:RedisClusterTlsCertificateFileName:RedisClusterTlsKeyFileName:RedisClusterCACertificateFileName:RedisClusterUsername
Table 3. Redis-cluster-plus-plus Configuration Field
Required
Description
Default
RedisServerNameOrIPAddress
X
Server name or IP address
Port
X
Server network port
RedisClusterPassword
Database or user password
ConnectionTimeoutValue
Connection timeout (in seconds)
3
UseTLS
1 = TLS; 0 = no TLS
0
RedisClusterTlsCertificateFileName
TLS certificate filename
RedisClusterTlsKeyFileName
TLS key filename
RedisClusterCACertificateFileName
TLS CA certificate filename
RedisClusterUsername
Redis user
The following is an example of four redis primary and replica nodes:Machine1:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername Machine2:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername Machine3:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername Machine4:7001:MyRedisPassword:7:1:/home/user/my-redis.crt:/home/user/my-redis.key:/home/user/my-redis-ca.crt:MyRedisUsername
Configuration file method
For configuration files, the data store type must be specified in the first uncommented line. Store-specific fields are provided on subsequent lines.
The following is an example config file for the redis type:
# This file is a simple configuration file for the Streams DPS toolkit.
# Any line starting with # is a comment line.
# Specify the name of the data store in the first non-comment regular line of this file
redis
#redis-cluster
#redis-cluster-plus-plus
# Specify the store-specific fields servers' names or IP addresses, port numbers, passwords and other fields as applicable.
#redis:
# RedisServerNameOrIPAddress:Port:RedisPassword:ConnectionTimeoutValue:UseTLS:RedisUsername
Machine1:7001:MyRedisPassword:7:1:MyRedisUsername
#redis-cluster:
# RedisServerNameOrIPAddress:port:RedisClusterPassword:RedisClusterUsername
#redis-cluster-plus-plus:
# RedisServerNameOrIPAddress:port:RedisClusterPassword:ConnectionTimeoutValue:UseTLS:RedisClusterTlsCertificateFileName:RedisClusterTlsKeyFileName:RedisClusterCACertificateFileName:RedisClusterUsername
A full copy of the configuration file can be found in any of the DPS sample applications.
App config method
For application configurations, the data store type must be specified with property key dps.nosql.db and each store-specific config with property key dps.serverN.
The following is an example of how a DPS application configuration named dps-cfg-for-app1 would be created using streamtool:
streamtool mkappconfig --property dps.nosql.db=redis dps-cfg-for-app1
streamtool chappconfig --property dps.server1=MyMachine1:6379:MyPassword dps-cfg-for-app1
streamtool chappconfig --property dps.server2=MyMachine2:6379:MyPassword dps-cfg-for-app1
streamtool chappconfig --property dps.server3=MyMachine3:6379:MyPassword dps-cfg-for-app1
streamtool chappconfig --property dps.server4=MyMachine4:6379:MyPassword dps-cfg-for-app1