Configure your Lookup Manager to use data from database
Most telecommunication projects store history and master data in a database, but the required business logic uses a subset of the data only. The streams processing requests look up information for every tuple that passes the business logic composites, resulting in a large performance load on the system and database to handle database queries continuously during file processing. Therefore, you create a lookup repository for the data subset in the shared memory when the system rarely updates this data in the database (for example, once a day).
Here the application control composite sends command requests directly to the DBCheck Switch operator. This operator forwards the request to the command splitter composite, depending on the results of the DBStatusChecker operator. The following figure describes the process flow for this specific use case:
Before you begin
As the lookup repository uses a database as the data source, this use case requires an installed and configured database.
About this task
Configure the Lookup Manager application to use a database as input.
Procedure
- In the file <PathToYourApplication>/config/config.cfg, find the lm.file parameter
-
Disable CSV file input by setting the parameter to off as follows: lm.file=off
-
In the file <PathToYourApplication>/config/config.cfg, find the lm.db parameter
-
Enable database input by setting the parameter to on as follows: lm.db=on
-
In the file <PathToYourApplication>/config/config.cfg, find the lm.db.vendor parameter
-
To use the DB2 database, set the parameter as follows: lm.db.vendor=DB2
-
In the file <PathToYourApplication>/config/config.cfg, find the lm.db.connectionName parameter
-
To use one of the connections that are defined in connections.xml, set the parameter to the wanted connection name, for example: lm.db.connectionName=SAMPLE
-
In the file <PathToYourApplication>/config/config.cfg, find the lm.db.name parameter
-
To specify the data source name (DSN) of the target database, set the parameter to the wanted value, for example: lm.db.name=DBName
-
In the file <PathToYourApplication>/config/config.cfg, find the lm.db.user parameter
-
To define the database user, set the parameter to the wanted value, for example: lm.db.user=dbUser
-
In the file <PathToYourApplication>/config/config.cfg, find the lm.db.password parameter
-
To set the database password, set the parameter to the wanted value, for example: lm.db.password=dbUserPassword
-
Open the <PathToYourApplication>/LookupMgrCustomizing.xml configuration file for editing
-
Define your repository segment schemas
-
After you adapted your configuration files, compile your Lookup Manager to create an application that uses a database as input
CAUTION: If the parameters lm.db.name, lm.db.user, and lm.db.password are provided as compile-time parameters, their values are visible in the SPL files that are compiled from the mixed-mode SPLMM files. You can provide those three parameter settings with the Application Configuration from Streams Console or you can provide them as submission-time parameters, so you do not have to recompile your application whenever your database credentials change.
Customization and connections files
The entries in the LookupMgrCustomizing.xml file must correlate with settings in the connections.xml file, which the com.ibm.streams.db toolkit requires. The Name attribute of the DataSource element must match the name of the access_specification that is defined in the connections.xml file. This file defines the connection specifications, each containing a name attribute, for example SAMPLE, and all database queries including user_connection and native_schema settings. The name of the access_specification for the lookup processing on the repository segment must be the name of the segment, for example DimMaster1.
The name attributes of the ValueDefinition elements in the LookupMgrCustomizing.xml file must match the name attributes of column element definitions that are used in the naming_schema element of the connections.xml file.
The SPLType attributes of ValueDefinition elements must follow the type definition mapping that the com.ibm.streams.db toolkit of corresponding settings in ValueDefinition elements and column elements describes. The following figure shows the cross-references between the customizing file, configuration settings, and input source:
Database configuration
The Lookup Manager supports two brands of databases, DB2 and Oracle, and requires the environment settings for the implementation and the driver support. For more information about these settings, see the com.ibm.streams.db toolkit documentation. If you use the DB2 database as repository source, you must source the DB2 profile.
The Lookup Manager uses the DBStatusChecker operator, which requires UnixODBC to be installed, in addition to the DB2 or Oracle database. All information that concerns the installation and configuration of UnixODBC is available in the com.ibm.streams.db toolkit documentation and on the unixODBC Project home page.
The following table identifies the required environment variables for DB2 and Oracle databases:
|
Environment variable |
DB2 value |
Oracle value |
|---|---|---|
|
UNIXODBC_HOME |
<path-to-unixodbc> |
<path-to-unixodbc> |
|
ODBCSYSINI |
<path-to-odbc.ini> |
<path-to-odbc.ini> |
|
STREAMS_ADAPTERS_ODBC_DB2 |
yes or 1 |
unset |
|
STREAMS_ADAPTERS_ODBC_ORACLE |
unset |
yes or 1 |
|
STREAMS_ADAPTERS_ODBC_INCPATH |
<db2-path>/include |
$UNIXODBC_HOME/include |
|
STREAMS_ADAPTERS_ODBC_LIBPATH |
<db2-path>/lib64 |
$UNIXODBC_HOME/lib |
|
ORACLE_HOME |
unset |
<path-to-oracle-home> |
|
PATH |
$UNIXODBC_HOME/bin:<path-to-db2>:$PATH |
$UNIXODBC_HOME/bin:$ORACLE_HOME: $ORACLE_HOME/bin:$PATH |
|
LD_LIBRARY_PATH |
$UNIXODBC_HOME/lib:$LD_LIBRARY_PATH |
$UNIXODBC_HOME/lib:$ORACLE_HOME/lib: $LD_LIBRARY_PATH |