Generating extended output data for databases
The framework implements file to file processing. If you want to have CSV output files that contain data that can be used to add data to the tables in a database, you need the table-based output type in your application. When you evaluate enrichment data in your custom PostDedupProcessor or in CustomContext the table schema needs to be extended, you need the extended table-based output type in your application. As the records of one input file can end up in different tables, one input file produces many output files in this use case.
About this task
Configure the application to produce extended table-based output data files per each input file.
Procedure
- In the file <PathToYourApplication>/config/config.cfg, find the ite.storage.type parameter description
-
To select the table-based output type for your application, set the parameter as follows: ite.storage.type=tableFile
-
In the file <PathToYourApplication>/config/config.cfg, find the ite.businessLogic.transformation.outputType parameter description
-
To select the extended table-based output type for your application, set the parameter as follows: ite.businessLogic.transformation.outputType=extendedTableStream
-
In the file <PathToYourApplication>/config/config.cfg, find the ite.storage.tableNames parameter description
-
Assign the names of the tables that are used by your application to the parameter, for example: ite.storage.tableNames=VOICE_CDR,SMS_CDR
Then, another application uses the output files, for example to load the content of these CSV files into a database.
The extended output stream (<namespace>.streams::TypesCustom.ExtendedTableStream or <namespace>.streams.custom::TypesCustom.ExtendedTableStream) becomes the input of the TableRowGenerator, which forwards the attributes of the ExtendedTableStream to its output. A tuple contains a single table row and one hash code for deduplication. If an input record results in multiple table rows or input to different tables, the Transformer must send the appropriate number of tuples.
The TableFileWriter uses the table names that you configure. For each table name, a dedicated spl.adapter::FileSink operator is used.